Managed Backups Failure

I haven't been able to successfully run a Managed Backup since 12/24. When I trigger a manual full managed backup, it takes a little over an hour before I get the attached error message: <empty response>

  • Adam Czajczyk

    Hello Matt,

    I hope you're well today and thank you for your question!

    I accessed your site and changed the logging settings for Managed Backups (so please do not change them). Could you please now enable WordPress debugging? To do this, add following lines to the "wp-config.php" file of your site:

    define('WP_DEBUG', true);
    define('WP_DEBUG_LOG', true);
    define('WP_DEBUG_DISPLAY', false);

    Make sure that this lines are above the "/* That's all, stop editing! */" line.

    Then please wait until the next scheduled backup time passes by (+ an hour or two) and download the "debug.log" file from your server from "/wp-content/" folder, rename it to "debug.txt" and attach to your response here.

    I will then review that file and will also access your site again in order to check internal Snapshot's logs. Hopefully this will give us a clue on what's going on there.

    Best regards,
    Adam

  • Adam Czajczyk

    Hello Matt,

    I checked the debug.log, the snapshot log and the site itself again and I think we should start with some basic adjustments.

    First step would be to raise the memory limit that WordPress is allowed to use. To do this please add following lines to the "wp-config.php" file of your site:

    define('WP_MEMORY_LIMIT', '256M');

    This line should be put above the "/* That's all, stop editing! */" one.

    Second step would be to instruct Snapshot to process backup using smaller "chunks". This will increase the time that it takes to complete but it often helps in cases like this and lets the script fully process the data. To do this add following line to the "wp-config.php" file (also above the "/* That's all, stop editing..." line):

    define('SNAPSHOT_FILESET_CHUNK_SIZE', 10);
    define('SNAPSHOT_TABLESET_CHUNK_SIZE', 50);

    Once this is all done, please wait for another scheduled snapshot and then check again if it completed. If it didn't please download the "debug.log" file again and attach it here just like you did recently.

    Best regards,
    Adam

  • Predrag Dubajic

    Hi Matt,

    I wanted to try few more tests on your site but unfortunately our support access has a limited time access and the session timed out before the snapshot could finish its thing.
    Could you send me in WP admin login details so I could perform further tests?

    You can send us your details using our contact form https://premium.wpmudev.org/contact/#i-have-a-different-question and the template below:

    Subject: "Attn: Predrag Dubajic"
    - Site login url
    - WordPress admin username
    - WordPress admin password
    - Link back to this thread for reference

    Best regards,
    Predrag

  • Adam Czajczyk

    Hello Matt!

    Two days is indeed very long but the question is whether it's a backup that's taking so long or has just hung up or is it the plugin that's giving a false information.

    You should be able to stop currently running backup by simply going to the "Snapshot -> Managed Backups" page and clicking the "Deactivate" button in "Backups Schedule" section. After that you may enable the schedule again and the backup should be stopped while the next run should go as scheduled.

    Before you deactivate it though please check the "View your log file" on the same page and share a couple of last lines with us here so we knew what was last activities performed by the plugin. Please also take a look at the "Backups" section for your site in The Hub (your WPMU DEV account) and see if any backup has appeared there.

    Update us please with that information.

    Kind regards,
    Adam

  • Matt

    Hi Adam Czajczyk here are a few lines from the log. I haven't seen a successful full managed backup since the manual backup they ran on 1/4.

    [Cron][2017-01-07 04:04:53][Info] Next start action scheduled for Sat, 07 Jan 2017 10:00:00 +0000
    [Remote][2017-01-07 04:04:53][Info] Use non-expired local token
    [Remote][2017-01-07 04:04:53][Info] Successful remote response for register-settings
    [Cron][2017-01-07 04:04:55][Info] Next start action scheduled for Sat, 07 Jan 2017 11:00:00 +0000
    [Remote][2017-01-07 04:04:55][Info] Use non-expired local token
    [Remote][2017-01-07 04:04:56][Info] Successful remote response for register-settings
    [Cron][2017-01-07 11:10:40][Notice] Backup ready to start
    [snapshot][2017-01-07 11:10:40][Info] Starting backup
    [snapshot][2017-01-07 11:11:42][Info] Created the backup and added the queues
    [Cron][2017-01-07 11:11:42][Info] Next process kickstart action scheduled for Sat, 07 Jan 2017 11:16:42 +0000
    [Cron][2017-01-07 12:45:41][Warning] Immediate hook misfired, kickstart backup processing
    [Cron][2017-01-07 14:37:15][Warning] Immediate hook misfired, kickstart backup processing
  • Predrag Dubajic

    Hi Matt,

    Thanks for the info, however these warnings shouldn't be affecting backup creation so we're dealing with something else here.

    Could you provide us with FTP or cPanel access so we can dig deeper into this?

    You can send us your details using our contact form https://premium.wpmudev.org/contact/#i-have-a-different-question and the template below:

    Subject: "Attn: Predrag Dubajic"
    - Site login url
    - WordPress admin username
    - WordPress admin password
    - FTP credentials (host/username/password)
    - cPanel credentials (host/username/password)
    - Link back to this thread for reference
    - Any other relevant urls

    Best regards,
    Predrag

  • Predrag Dubajic

    Hi Matt,

    Thanks, I did few more tests and the manual backup still seems to be working for me, you now have backup from today.

    However I'm quite lost about automatic backups so I'll need to forward your login info to our developers so they can have a look at this and see what's causing this strange behaviour.

    Please note that, developer response might be slower than usual staff response, so we appreciate your patience on this.

    Best regards,
    Predrag

  • Adam Czajczyk

    Hello Matt!

    I'm sorry for keeping you waiting that long. We had to pass that to the developers since we weren't able to solve that here and it may take them a bit longer to respond than it usually takes us here on forum.

    However, meanwhile I would like you to run two additional tests. First, please add following line to the "wp-config.php" file (above the /* That's all, stop editing... */ line):

    define('SNAPSHOT_FILESET_USE_PRECACHE', true);

    and try running backup again to see if it completes. If it doesn't, in addition put yet another line in the "wp-config.php" file (also above the "/*That's all, stop....*/ one):

    define('SNAPSHOT_FILESET_LARGE_FILE_SIZE', 104857600);

    then make sure that logging on "Managed Backups" page is set to "Info" for all sections, run backup again and after it fails again see the last lines of log on "Managed Backups" page. There should be some additional information so share these last lines with us here please.

    Kind regards,
    Adam

  • Lindeni Mahlalela

    Hello Matt.

    I hope you are doing great today. As I was going through old threads for maintenance, I noticed that this one is still open and we haven't had any response from you since my colleague Adam made a suggestion as means to solve the problem.

    I was wondering if you still need help with this or you got it resolved by applying the changes suggested by Adam or by other means. We will be happy to know you got it resolved and we will be more than happy to help resolve it if you are still having the same issue, so just please confirm if you still need help.

    I hope you enjoy your day further.

    Cheers,
    Mahlamusa

  • Lindeni Mahlalela

    Hello Matt

    I hope you are doing great today and I am sorry that this is not working out for you. I understand the frustration of not having a remote backup for your website, you may always think about what will happen if you lose the local backups of your site.

    I went through the setup, I have tried it on my own server and it worked perfectly but failed on your server. Looking at the logs written on the file 'wp-content/debug.log' I have found the following error as a highly probable cause for the backups not running.

    PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /home2/f1r1x5q5/public_html/wp-content/plugins/snapshot/lib/Snapshot/Helper/Zip/Archive.php on line 33

    This means there is a PHP timeout when executing the Archive.php file which is responsible for zipping the files before they are sent to the remote server. The backups are executed but Snapshot fails to zip the files into an archive. The logs shown in the page "Snapshot > Managed Backups" are those written by the plugin itself to show what steps it executed and are different to those written by PHP in the log files.

    To solve this issue, I have changed the maximum execution time for your website through cPanel by going to 'Software > PHP Selector" then click on the "Setup custom php.ini" button, on the form I have changed:
    max_execution_time = 1800
    This means PHP will be able to run for as long as 1800 seconds instead of the default 30 seconds which was timing out, this means now that the Archive.php script will be able to execute for as much as 30 minutes, which I think is enough. In most cases, execution time this high is not required, in my test server I use 120 seconds and I get no errors at all, this also depends on the server itself. You may change this to a lower value as it is not recommended to allow scripts to execute for a very long time. You may try to set it to 5 or 10 minutes and see if it completes the backup, if it does then lower the value gain.

    After applying the change, backups should run smoothly and you should see them listed in the page "Snapshot > Managed Backups" and also on The Hub on the Backups page available by clicking the icon that looks like a cloud with an arrow pointing upwards next to the domain of the website.

    I have renamed the old 'wp-content/debug.log' file because it was too long and hard to read, there is a new one now and this does not have the PHP Fatal error mentioned above. I kept both files in case you need them.

    I hope this helps and if you need more help or assistance please let us know and we will be more than willing to help in anyway possible.

    Cheers,
    Mahlamusa

  • Lindeni Mahlalela

    Hello Matt

    I forgot to mention that if the backups fail again then you should check the error logs and see what is there, if it still shows the timeout error then you should go to your cPanel account in 'Software > PHP Selector" and click on the "Setup custom php.ini" button, find the 'max_execution_time' and set to much higher value like 3600 for an hour or anything of your choice.

    I hope that helps.

    Cheers,
    Mahlamusa

  • Lindeni Mahlalela

    Hello Matt

    I hope you are doing great today. We are still working on this issue and there is something we need to check on your network admin but unfortunately support access has expired and when we try the login details you sent to my colleague Predrag, we get the following error:

    You attempted to access the "FitPro Site" dashboard, but you do not currently have privileges on this site. If you believe you should be able to access the "FitPro Site" dashboard, please contact your network administrator.

    This means the user does not have admin privileges. Could you please give the user super-admin privileges or grant Support Access again. We will continue with the investigation and hopefully solve this soon.

    I hope to hear from you soon. Please enjoy the rest of your day.

    Cheers,
    Mahlamusa

  • Lindeni Mahlalela

    Hello Matt,

    I am really sorry about the delayed response here, I have been working on this issue and I was able to change the server settings on your cPanel and done some tests, it seems we are getting closer and closer.

    Here is a summary of what I have done and why:
    1. Firstly the site was hitting PHP memory and execution time limit. All attempts done to php.ini and wp-config.php were overlooked by the host and were not effective. So I raised the limits on cPanel "Software > MultiPHP INI Editor", I selected the domain, and the PHP version being used and raised the limits there.

    After that the limit related issues were gone but still backups were not completing.

    2. I disabled WP Cron and added a server side Cron Job that will run and call wp-con.php, this is more reliable than WP Cron

    in wp-config.php, I added the line: define('DISABLE_WP_CRON', 'true');

    Then in cPanel "Advanced > Cron Jobs" I added a cron job that runs the following command:

    curl -I http://yourdomain.com/wp-cron.php?doing_wp_cron

    I verified if the cron runs by making it send an email each time it runs,

    3. In wp-config, the following lines were already present:

    define( 'WP_MEMORY_LIMIT', '256M' );
    define('SNAPSHOT_FILESET_CHUNK_SIZE', 10);
    define('SNAPSHOT_TABLESET_CHUNK_SIZE', 100);

    At this point, I was able to run a Managed Backup and it seems to have completed. Can you please check on your site and on The Hub to see if the Backup was pushed to the remote server or not. Please let me know if you get it there or not.

    We are still working on this and as it seems we are making progress but we need more time and a few more tests as this looks like a complex issue. There is a lot that we have to do on the server side on cPanel to get the Snapshot to work.

    Unfortunately, for further tests we still need to have access to your server. I urge you to raise the privileges of the account you have created for us because when Snapshot is running for a long time, Support Access logs out automatically and we have to start over again. This is not good as most of the time it logs out after an hour or so and that means the hour is lost in waiting. Please update the account you have created for us so that we can have super admin access. You can remove the account once we are done with our tests.

    We really appreciate your patience as we try to resolve this and I hope we get done with it soon.

    Please enjoy the rest of your day.

    Cheers,
    Mahlamusa

  • Lindeni Mahlalela

    Hello Matt,

    Thank you for granting access again. Just to assure you, all my tests are not affecting any user sites and I will make sure to get things back as they were if I did some major change, but none so far. I have ran a few manual managed backups, it seems they a re running as they should but just that it takes longer.

    The only issue I am facing right now is that Support Access has a time limit per session so I get kicked out before I complete a backup. But this time they are much faster than before. I need you to go to your website's dashboard on the page "Snapshot > Managed Backups" you will see a couple of new backups check their names and then go to The Hub in the backups tab and see if they were uploaded or not. At this point I see that not all are uploaded to our remote servers but I think that is a matter of time.

    On the managed backups page it says 4GB / 10GB is used up which gives me the suggestion that the files are being uploaded, please double check on The Hub and let us know if all looks good. I would continue to run a few more backups but Support Access times out before I can finish and that results in lost time as it will start over even if it timed out at 99% like the last time I tried.

    So please try to run the backup from your side and see what happens, if it gives you some trouble and you want me to continue some tests on my side then please send WordPress super admin login details, you may raise the privileges of the previous account you created for us or create a new one with super admin access.

    I am looking forward to hearing back from you.

    Cheers,
    Mahlamusa

  • Matt

    Hi Mahlamusa - so I got an alert from the WPMU Dev Uptime Monitor this morning at 12:52AM saying that my production website was down.
    [image pos="0"]

    Talking with Tech Support for my hosting provider they were pointing their fingers towards the automated backups and mem setting changes as the root cause.

    When I checked the Managed Backup settings, I was presented with the below screen asking me to enter my key again, instead of showing my the managed backup settings page.

    Can we adjust the backup settings so it won't take down my production site the next time it runs a full managed backup?

  • Lindeni Mahlalela

    Hi Matt,

    I hope you are doing great today and I am very sorry for the inconvenience caused by the down time on your site and I hope it does not happen again.

    Talking with Tech Support for my hosting provider they were pointing their fingers towards the automated backups and mem setting changes as the root cause.

    The memory settings on your site should not be a problem here, they are set to a recommended 256MB which is fair for most shared hosting even the low end ones. If your host says this is an issue then they have set their limits too low. WordPress comes with a default 64MB memory limit suitable for the default wordpress installation, but once you add some heavy plugins that becomes an issue and you will receive errors, that is why the memory is currently set at 256MB.

    When I checked the Managed Backup settings, I was presented with the below screen asking me to enter my key again, instead of showing my the managed backup settings page.

    We are currently experiencing issues with our servers, maybe this is related to that but I doubt because even if we are experiencing issues your settings should stay intact. Just to confirm, please try to get a new Snapshot key and enter it on your settings and see if the plugin will accept that and show you your snapshots. Please set your key and visit the hub and see if you can see any backups there, let us know what you find.

    Can we adjust the backup settings so it won't take down my production site the next time it runs a full managed backup?

    For now, I have decreased the max_execution_time to a lower value than what I set before, this is to make sure that scripts run for a short period to avoid high resource usage, but this limit must be set so that the plugins (Snapshot or any other) do not time out while executing. I have also adjusted the Cron Job to run twice per hour but this may cause snapshot not to complete on time, but it will ease on resource usage on the server.

    On your side, you can change the frequency of the managed scheduled backups to Weekly, I set it to daily for testing.

    I hope this helps. Please let us know what you find and we will be happy to help in anyway possible.