Unable to restrict media upload (PDF) by membership type

Hi there! I've set up 2 membership groups... Default and Restricted. I gave the Restricted Group access a particular media upload (a pdf), but i'm still able to see it when logged in as my test Default member. You can even see it in the Media Library. I must be missing something... Help would be appreciated!