Audio streaming to reduce server load

I'm building a membership site to host training video's. The video's are hosted on vimeo. Along with this, there are audio downloads. Are there any significant 'wins' in use of server resources by hosting the actual audio files external to the main website - for both streaming and download i.e. on a paid Dropbox or amazon account...or on a secondary hosting account Heart Internet Re-seller). At the moment, I've got the audio files (MP3) within a folder of the main site, but I know it's going to be high-traffic site and want to do all that I can (within budget) to keep the site running fast. It is on it's own dedicated server. Does anyone have any thoughts on this please?