How to speed up my site

Wasn't sure if I should open a new thread based on the one about performance issues, but my site is much smaller than those discussed, although it gets quite a bit of traffic.

The problem I keep on having is running out of memory. I have increased my memory on my server, but need something else to also boost my site. I know many plugins that I have installed eat the resources available.

I have WP Super Cache installed which works ok. But Been reading a lot of good things about W3 Total Cache, used on Mashable,com, and quite a few big sites.

Not sure if it works that well on WPMU though.

Friend of mine recons that Varnish is best and referred me to this reference for comparisons...

Then, memchache with W3 Total Cache?

Many solutions and combinations! So in your experience, what is the best way to do this?

I'm running on a VPS currently.