We have a site at http://www.themoneyprinciple.co.uk which is rather slow to load. It is on quite a skinny server at the moment but the main problem is both the size and response of Facebook. I realise that some minify is needed and other things. The theme was originally based on Artisteer but has been extensively modified. Feel free to look at the source but you will see what I mean.
There are individual FB like buttons on each post from the "Really simple Facebook and Twitter share buttons" and also a Fanpage box which shows pix of the fans. When I disabled this plugin on a development the download went down by a factor of about 3!
Analysing what is downloaded, each Like button is 24kB and the .css and .js alone for the Fanpage box (which is in an iframe of course) is quite large. Note I am not using FB login or connection or anything. I just want readers to be able to Like individual posts (and hence get them published on their pages) or join the MoneyPrinciple fanpage. By comparison, Twitter is lightness itself (but of course does rather less work).
I see that WPMU has the same box and Like button but is a lot faster. This may be because you are using a much bigger server or nginx (as you are) but then we do not have many sites on my server nor that many visitors. I also see that you are using Comments Plus but other than that how do you minimise the FB download time? It is a real irritation and is affecting our ranking I think.