How to speed up my site

Wasn't sure if I should open a new thread based on the one about performance issues, but my site is much smaller than those discussed, although it gets quite a bit of traffic.

The problem I keep on having is running out of memory. I have increased my memory on my server, but need something else to also boost my site. I know many plugins that I have installed eat the resources available.

I have WP Super Cache installed which works ok. But Been reading a lot of good things about W3 Total Cache, used on Mashable,com, Yoast.com and quite a few big sites.

Not sure if it works that well on WPMU though.

Friend of mine recons that Varnish is best and referred me to this reference for comparisons...

http://cd34.com/blog/scalability/wordpress-cache-plugin-benchmarks/

Then, memchache with W3 Total Cache?

Many solutions and combinations! So in your experience, what is the best way to do this?

I'm running on a VPS currently.

Thanks,

George