I disabled various plugins to find out what was slowing down my site load (on advise from the hosting team).
Turns out when I disable the Google Maps plugin, my site loads up way way faster (I don't know how to measure it programmatically).
How can it be optimised? It is the first thing my page loads, so the rest of the page doesn't show up until the map is done loading. I have it displaying as a widget.