I am having trouble getting the membership access

I am having trouble getting the membership access and denial of access working for BuddyPress groups. If I allow all groups it works fine, but if I want to restrict groups each membership level can have access to then there is no access to any group
I have tried doing both positive and negative settings and get the same response with either