What is the critical number of plugins for a blog network that grows at 10 to 20 subsites per day?

I will like to know when plugins become too many such that they begin to affect the Network speed. I have heard 12 to 15 and I have also heard 20. For my install because of numerous customizations I am worried I do not have much wiggle room. What will be a ballpark number to shoot for?