Can we not load CSS and JS on non-map pages?

Is there a way to not load google_maps_user.css and google_maps_loader.js on every page of my site?