E-Newsletter issue not responded to??

I installed the eNewsletter plugin, input the settings etc, then tested the settings, seems like the install went pretty smoothly. I then uploaded a CSV file with my contact list, all seemed to go well. Later that day five people subscribed to my newsletter too.
I then tried to mail out a newsletter, but the plugin would only send the email to the new subscribers and not the other people in my list.
Any thoughts?
Mark