Google Maps Directions Box Sometimes Doesn't Load

I'm working on a site where we have some maps that are placed in columns on the far right of a page, for example:

Something strange appears to be happening where the Directions box that usually shows up underneath the map often never loads with the page. Once in awhile it will show up.

Is this a known bug or could something else be going on?