Domain Mapping Response Time is Slow


I did a website speed test and the results show that the response time when using domain mapping plugin is very slow. There is a long wait time before requests are handled and returned which is hindering the overall site performance. Once the requests is returned the site loads quickly.

Note: I have an other software (non WP) which is on the same server (dedicated) environment and the response time is much quicker in the 3s mark.

Also I have a single WP install without domain mapping plugin on a smaller server and the response time is also quicker.

Please see image.

How can this be improved?