Logged out of front end when using SSL


When using SSL in the backend and front end I get logged out when visiting the website. If I switch to run the website through http it works fine. I never noticed it until recently I started running the whole multisite using SSL instead of certain pages like I had them before.

If I need to get back in to make some changed on the front end I simply go into domain mapping, hit the key to toggle to http and visit the website. Wait a few seconds and it logs me back in. Once logged in I go back to Domain Mapping and toggle it back to https.

Any ideas or suggestions?
The only new plugin that I have installed recently has been Hustle.
Support access is open in case you want to take a look. I left you a note over there just in case.