I found a potential problem with the pop3 connection test.
The test appears to work so that the plugin sends a test email to the bounce email box and then checks if it has arrived. There is no delay between sending and checking. This was not a problem one installation but I kept getting fails on another server. It turned out the email did get delivered fine to the mailbox but it took a moment. The test in the plugin fails because it checks too early. Adding
between sending and checking takes care of the problem. Would not hurt to have a delay there in case other people experience the same issue.