Closing The Message - Keeps Showing Regardless of Option

We are using the slide-in and we don't want the slide-in to keep showing, but no matter what option we choose in "closing the message", it still shows up. Any ideas?