I implemented the system shown in the following link into one of my sites. The site has about a dozen different videos on the page. When I was measuring the before and after speed, I WAS SHOCKED! It literally went from around 8 seconds to about 1.5s.
It seems rather silly that WordPress doesn't enable this functionality by default. I could see a few reasons why someone might not want it implemented, but the Pros definitely out weight the Cons.
Anyhow, here's the page that I referenced to implement the YouTube Video Defer. It's a rather simple implementation, but would become time consuming to set up on multiple websites.