What are your recommendations for running multisite networks?

Greetings,

I have a 4Gig, dual-core linux server with MySQL on it. I have installed two WP multisite networks and there are a a couple other WP sites on the server as well. I don't have any member sites yet on the networks as I am still setting up, testing, etc. I am using about 90 WPMU plugins and a few custom plugins including WP Super Cache.

I am already having performance problems and don't even any member sites yet. I have used P3 Performance Profiler and identified the worst performing plugins. The two worst are Ultimate Branding and MarketPress. Q&A is another, Infinite SEO and a few others WPMU plugins.

I am going to downsize to one network and get rid of some of these plugins as I can't scale up or out with servers at this time. Do you have any recommendations on server setup, performance ideas or anything else that could help me improve this situation?

thanks