How to make my site load faster

Does it make a difference on how quickly my site loads if I have my domain name sitting at go daddy and sent to hostgator, or would my site load faster if my domain name was actually on hostgator?