Im have difficulties getting my site to perform even

Im have difficulties getting my site to perform even moderately fast. At first I became a member or wpmudev. whoot whoot. Then I started installing a good majority of the plugins available thru the wpmu dashboard. Then my site became ridiculously slow. ie. 45-50s times via webpagetest.org. I was on shared hosting but switched to VPS, then tested and was slow. Dreamhost support was even shocked at the slowness.

Since Im still in development stage I just deleted everything and started from complete scratch again. Im using multi-db eventually (still have bugs).

My main question is this: I have a new multisite with a couple minimal plugins (akismet and a base plugin of my own to add functions separate from the theme's functions.php) and buddypress installed. Since installing buddypress I can already see a difference in my page loads. Site load is already slower along with anything in the admin. How can I speed up the site?

Im using Dreamhost VPS. My first go around (before complete do over), they said that I had too many plugins and needed to remove non essential plugins. Now with just buddypress I can already feel a hit. Is there anything you can recommend, or should I switch hosting providers? idk what to do.

I definitely need multisite, a Social Network, Events, Membership, Member Directory, Ecommerce, Gravity Forms, Fundraising, Classifieds, Pro Sites, eNewsletter. Probably more...

Is it possible to use almost all mpmudev's plugins and still have 5-7s page loads? If so what does this take? Do I need to spend 1000s/mo for a server that can handle all of mpmudev's plugins on one giant multisite? Any companies you recommend? ....

idk where to go from here? Anything would be helpful. Thanks