Snapshot plugin getting stuck and sucking up all bandwidth

Hello,

I am using your snapshot plugin on several of my websites, all hosted from the same server.

The snapshot plugin is running on 7 websites, and all the backup processors for each sites are set to run 2-3 hours apart from each other, as I don't want to start backing up all 7 websites at the same time, for obvious reasons.

The issue I'm having is that on 3 of the sites, when the backup process is complete (.zip created) the file is then sent to my Amazon S3 Server, all goes well for 4 of the sites, however for 3 of the sites the uploading process seems to get stuck, and then sucks up all my upload bandwidth from my server. The largest file from one of my wordpress sites once the .zip is create is around 900MB.

Something somewhere seems to get stuck, as my server becomes very unstable and the sites take a considerable amount of time to load, and in some cases doesn't.

I ran iftop to see what was taking all the bandwidth and it is the service uploading to amazon.

I tried to abort the uploading process through the admin interface of the snapshot plugin, however clicking the Abort option within the snapshot didn't work.

I then proceeded to delete each of the backup jobs from the 3 trouble sites. This seem to resolve the problem for 2 of the sites, however 1 site was still running the process and taking bandwidth, even though the job had been deleted.

Rebooting my server, with all Snapshot jobs deleted fixed this issue.

Is there anything I can do to try and resolve this for the 3 troubled sites?
Any way to debug this?

I have attached a couple of pictures from iftop showing the bandwidth being used.

If you need any further information please don't hesitate to ask. Many thanks for your help and support on this on.

Thanks.

  • Nicholas

    Hi Mrinal,

    Thank you for your reply.

    I have double checked this for you, and there are no white spaces.

    I'm currently testing each individual site with the SnapShot plugin to see where to problem may lie. As I'm testing this across 7 sites, its taking a little time.

    I think what was happening, is that while a backup was already in the process of uploading to Amazon, another backup starts and then tries to transfer to amazon at the same time, and something somewhere is getting either confused or stuck in the process.

    To keep you updated, I'm currently testing each site indivdually, with only one site backing up a day, to give each site enough time to backup and send the file to amazon, even though 24 hours is a little excessive, I want to make sure the sites which were having the problems still have the problems if only that site is backed up and then sends the file to amazon.

    Once I see all sites have backed up individually - where only one site a day is backed up, I will then try splitting the backups into two days (3 one day - 4 the other) rather than backing up 7 sites in one day with only a couple of hours apart.

    As these sites are live, I can only test in the evenings, as for some of the larger sites the transfer process to amazon does take up quite a bit of bandwidth.

    While I'm doing all of this, I wonder if you could answer a question;

    Do you know if it is possible to limit / restrict the amount of bandwidth used for Snapshot or the transferring of the .zip files to amazon?

    Many thanks for your help and support.

    Cheers.

  • PC

    Hey there,

    Thanks for posting back.

    I think what was happening, is that while a backup was already in the process of uploading to Amazon, another backup starts and then tries to transfer to amazon at the same time, and something somewhere is getting either confused or stuck in the process.

    That is quite possible. Did you try running only one backup at a time ?

    Please note the backup process is like, a local archive is created and then its uploaded to amazon (or anywhere else)

    Please keep us posted of your test results :slight_smile:

    Do you know if it is possible to limit / restrict the amount of bandwidth used for Snapshot or the transferring of the .zip files to amazon?

    I am not sure for an answer on that and would like to call in our lead developer @Paul for his expert advice.

    He should be around soon !

    Cheers, PC

  • Nicholas

    Hi PC,

    You're gaining a loyal follower here :slight_smile:

    That is quite possible. Did you try running only one backup at a time ?

    Yes, I ran two sites separately (Two different evenings), and no issues.

    As I have 7 sites and can only backup during the evening when there is less traffic to the sites I try to spread the backups around 2-3 hours apart. I believe that the backups were catching each other up, especially some of the larger files being uploaded to amazon - Again still testing :wink:

    Thanks for your help @Mac

  • Nicholas

    Hi Ari,

    Sorry for the late reply.

    I have finished testing each site separately, and when backing up to amazon with one site uploading daily no problems.

    Two of my sites are only 150MB and take no time at all to backup (.zip) and upload to amazon so I do these two sites on the same day (several hours apart).

    As for the bigger file sizes I've had to allow each site a single evening to backup.

    Upside - no bandwidth loss or stuck uploading processes

    Downside - can not backup the larger sites daily (as the upload process to amazon seems to get stuck when two of the larger files get push to amazon at the same time).

    If I only use local backups, there are no problems. The problems only comes when uploading to amazon and two large files are sent to amazon at the same time.

    Is it possible to add anything on a server level to hold back from uploading to amazon, if an upload is already in process?

    Can snap-shot be told to only use a certain amount of upload bandwidth, a similar feature you may find in torrent clients.

    Thank you for your help and support Ari, really appreciate it.

    Regards.

  • Nicholas

    Hi Ari,

    Thank you for your help.

    Yes, this is a little strange. I run the same plugin on another server and no issues.

    However one this server, even the smaller backups swallow all the server bandwidth until the upload process to amazon is complete.

    While the file is being uploaded to amazon the website are very slow and in many cases when dealing with some of the larger files uploading, non responsive.

    I also use PINGDOM to monitor my sites on the server, and when snapshot uploads the filed to amazon I receive a notification stating my site has gone down.

    Now I know the site isn't actually down, but due to slow loading times while the backup file is being uploaded, the latency is very high, so pingdom reports the site as down.

    I hope this gives you a little more insight, anything I can do to help please don't hesitate to let me know.

    Many thanks.

  • Dharmendra

    Hello Nicholas,

    I hope you are well today and sorry to hear about the issues you are facing.

    I have reviewed the issue thoroughly. Currently, controlling the bandwidth limit wouldn't be possible out of the box as it will require lots of custom coding in the plugin. However, it is possible to control the bandwidth limit via network firewall if it supports the advance configuration.

    You can ask your network administrator for throttling the bandwidth limit with Amazon S3 Server and your host. The limit depends on the network port speed and average network traffic of your host. So you can change the limit and review the result.

    I hope that helps. Please feel free to ask if you have more questions on the same.

    Have a great day! :slight_smile:

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.