Community: Would you cache a site for 1 sec per page savings?

This is a community question. Unless this site no longer really is about that anymore

But who here would worry about caching a site to only save 1 second of load time.

Example, with compression on (mod deflate) - and caching after load set, I see a 2.5 - 5.0 per page load depending on speed of connection (1.4mb page, which yes, needs to be optimized)

However it seems that currently my database is only adding a second or so to my queries. When i saved the source of my homepage. My site loaded in 1/2 the time.

So my database calls really arent that bad. I don't run a lot of widgets, etc... its just my damn headers.

But still - google has reported load times of 10 seconds for my users when my vps is running a little slow (and many are only slow DSL rural types of connections)

So - would you install a caching option. I do have multisite, and so if your answer is yes, got a recommendation for that?
(i dont want to waste all my space saving 400 pages of poorly optimized sites)