Comments added to cached pages are only visible for logged in users

I've noticed multiple times (it's been going on a while now) that, on my multisite, when comments are made the cached pages are not refreshed. People not logged in will keep seeing stale pages. I've noticed it on multiple sites in the network.

I've had the Hummingbird debug logging switched on for about a week now but I don't see any log file appear anywhere...