WP Super Cache Anomaly?

We are finishing our migration of clients from static sites to MU and have Wp Super Cache installed and working with no issues except for one weird scenario that may or may not be related to caching.

One client was experiencing 10 to 20 second load times even though we saw load times as being fine and couldn't reproduce the issue.

We had the client go to other sites on the server that were static and not on mu and he had no load time issues and when he went to different sites on the same server on MU, the page load time issue occurred consistently.

I turned off wp super cache and the user no longer sees the load time issues on MU.... has anyone heard of caching causing this type of issue? very strange, but it concerns me because it opens up the possibility that it is occurring with other users if this isn't some super strange anomaly. Is it possible for users to have longer load times when accessing a cached page given the right circumstances?