Move plugins to a different server

I'm wondering if this is possible. I'd like to host my plugins folder on my CDN server in an attempt to de-bulk my multisite. Is this possible to do, also, would it have any affect on speeding up the load time?

Thanks,
Sam