Is there a suggested number of plugins & themes to keep up the speed of a multisite?

I'm creating a new multisite and am having great fun adding a large number of plugins from the Multisite section on here, but if I add a lot of plugins, will this cause a problem with speed on the primary site?

Most of these won't be activated on the main site.