Is this correct annoying behavior for multisite?

I log into my main Network site as a super-admin user. I then go to one of my subsites and it requests a login again. In both instances, I select remember me when logging in.
If I go from from the subsite dashboard back to my Network dashboard, it loads right up. When I then go from the Network dashboard back to the subsite I was just at, it asks me to login again.

Is it supposed to work that way or is there a setting somewhere I can turn on so I do not have to keep logging into the subsites?