Browser has to connect to each sub-site before loading admin pages

I've got a Multisite I've been running a while, and it's started getting these long delays before loading pages of the admin. I noticed that the browser says "Waiting for" then it runs through each sub-site/domain on the network, before completing the load of the admin. With 40 or 50 sites on the network, it takes a long time to do that. I could see it needing to do that on certain pages of the network admin, but it does it about 75% of the time on any admin page, including the post editor of individual sub-sites. It's really slowing me down.

I don't have time (yet) to go through and disable all plugins and try to narrow it down to 1... assuming a plugin is the problem at all. So I'm wondering if anyone has seen this before? Any ideas what's causing it, or how I can tell why/what is being called from each sub-site?

It's really annoying. I don't know if this is related, but I was in phpMyAdmin the other day and noticed that for some reason the wp_blogs table does not have all the sub-sites in it. The last 5 or 10 are not in there. I have no idea if this could be related, but thought I'd mention it just in case.