Wordpress Multisite became slow and /tmp folder rapidly growing

I am running a Wordpress Multisite with around 50 websites on it. Since a month or so the server is becoming very slow. The /tmp folder on my website is also rapidly growing, there are lost of Wordpress sessions in it. My best guess it that this is a result of the double sessions that are created by the Domain mapping tool. (I have Cross-domain autologin enabled).

I have load times of 9 seconds and sometimes more. With hummingbird I already tried to improve the preformance, with little succes. GZIP, Image optimization and cache are all on. Also I ran a database Optimizer / transients cleaner.

https://tools.pingdom.com/#!/dOtxZu/webcademy.nl

If you look at the waterfall there are mutliple long wait times of 2.42s, 3.07s and 5.06s. I am running on a VPS with 4GB RAM, SSD storage and 2 intel Xeons. There are no websites on the server, other then the 50 websites that are inside the multisite.

Do you have any idea where this long loading times are coming from? Is there something I can do? We would like to scale up to 100 and even 250 websites inside the network. Is this possible with a multisite setup?

Thanks a lot!