Can Changing DNS A Record to point at another VPS IP (but same site) slow a site down


We currently have the same site on a reseller and now a VPS server… we are about to change the DNS to point at the VPS. In the process of doing this before we noticed a negative change in the speed when it should have gone up… is this possible or should changing dns not effect the speed (especially when it was pointing at a faster server)?