How do you handle your newsletter?

Quick question, maybe I am making a rookie mistake here but bear with me:

I send out a newsletter to about 1000 recipients from: newsletter@mydomain.com. That is also the replyto email which is monitored by my newsletter system to process bounces.
Unfortunately there are plenty of people who simply click reply and reply to this email which I do not monitor as it tends to fill up with countless out of office replies and I don't want to interfere with the bounce system anyway...

What do you do? How do you handle this? Sometimes there are essential mails in this inbox :disappointed:

Should I use i.e POP3 to check this inbox without marking anything as read or deleting so as not to disturb the bounce system?