Domain Mapping slows everything down

Whenever I add a domain to be mapped to a blog, the pages start taking a really long time to load until the dns is updated and the domain actually points to the correct server.

I have selected different options for access to the admin area (under Super Admin > Options > Administration mapping) but while the domain is not directing people correctly (due to dns propagation), except for when I select "Mapped Domain", everything still works but pages take forever to load.

Any other people having the same issue? Is there a fix for this or do people have to wait for propagation?