Prevent slide-in to show up on page

Though I checked "Do not show a slide-in on this page", a slide-in shows up on all pages. Did I miss any settings? Or does the "do not show" option only works with posts?