When I create a new site on my network I then go in and create a new user for that site....

When I create a new site on my network I then go in and create a new user for that site. I just noticed that when I create a new user for a site, that user is added to ALL of my sites!
That was not expected, and not what I want.
The only users on each site should myself as super admin, and the new user/owner of the site.
Is there a toggle switch or something that I am missing that needs to be turned off?
It is getting a bit hectic, as my network grows, to go through each and every site each site I create a new site and user, and remove that new user from the other sites!
Thanks.