Snapshot Failing to Run Backup

Hi,

I am using Snapshot Pro on this multisite installation to backup all the sites on the Network. I have a backup scheduled every week for each site, including the main site. Each backup is running fine with the exception of the main site, which appears to start and stop very quickly when run either on schedule or using the manual “run now” link.

From my PHP error logs, each time it attempts to run, I get 2 errors, one right after the other:

mod_fcgid: read data timeout in 45 second

Premature end of script headers: wp-cron.php

Can you please help with this? I have enabled support access. Please feel free to try and run these / change the snapshot settings, but please don’t mess with any other plugins or settings as this network has live sites running on it.

Thanks,

  • Predrag Dubajic
    • Support

    Hey @dominic,

    Hope you’re doing well today :slight_smile:

    Could you try increasing your memory limit and maximum execution time to see if that will make any changes?

    You can increase the WordPress memory and maximum execution time by following the instructions provided on below pages.

    http://docs.woothemes.com/document/increasing-the-wordpress-memory-limit/

    http://codex.wordpress.org/Editing_wp-config.php#Increasing_memory_allocated_to_PHP

    http://codex.wordpress.org/Common_WordPress_Errors#Maximum_execution_time_exceeded

    Let us know how it goes.

    Best regards,

    Predrag

  • Predrag Dubajic
    • Support

    Hi Dominic,

    I tried running one snapshot without media files and it was created without any issues, next one I tried creating with media files included and it failed at iThemes security.

    Could you try temporarily disabling that plugin and run the snapshot then.

    You should also check if there are any rules in .htaccess and in wp-config.php created by iThemes Security plugin and try removing those so we can know if issue is caused by that.

    Best regards,

    Predrag

  • Dominic
    • The Incredible Code Injector

    Thanks Pradag,

    I’ve tried a couple of things, one – disabling iThemes Security as suggested, and second, I added /www-data/wp-content/plugins/better-wp-security to the exclusion list in the snapshot, but both produced the same error again…

    How did you manage to see where the backup process had failed?

    Thanks,

  • Predrag Dubajic
    • Support

    Hi Dominic,

    Once you run snapshot process it will show what is currently doing and when it stopped for me it was at iThemes plugin.

    I couldn’t run it again to attach you the screenshot since support access has expired, could you grant it again so I could try running few more tests and see if I need to call in our devs on this one?

    Best regards,

    Predrag

  • Predrag Dubajic
    • Support

    Hey Dominic,

    I believe I have found what is causing it, the Snapshot is actually failing at iThemes Security backup folder, I misread the path last time, after I excluded /wp-content/uploads/ithemes-security/backups from snapshot it was able to finish normally.

    Can you tell me what is the size of /wp-content/uploads/ithemes-security/backups/ folder?

    Best regards,

    Predrag

  • Dominic
    • The Incredible Code Injector

    Thanks again Pradag, makes sense it would be failing there, however adding that to my snapshot doesn’t fix the issue. I also compared the test snapshot you created, the other other differences I can see are – you had manually selected files where I had clicked all, and yours was local only, where mine is set to send to Google Drive. I’ve tried changing all these settings and it still will not run. Additionally, I can’t seem to run your backup either…

    Have re-enabled support access, and would appreciate you showing me how I can view the logs as I think this will help me do additional investigation myself.

    Thanks,

  • Predrag Dubajic
    • Support

    Hi Dominic,

    I just changed my snapshot to include all files and kept the excluding folder mentioned before and the snapshot was still created successfully.

    Can you try editing my snapshot yourself and try selecting destination for it once to GDrive and once on DropBox to see if there is maybe something not configured correctly for GDrive?

    Best regards,

    Predrag

  • Dominic
    • The Incredible Code Injector

    Hi Pradag,

    Before trying your suggestion, I’ve just tried running your snapshot as is and it’s STILL failing for me, which I find extremely strange – surely there can’t be an issue based on the user who is triggering the snapshot??

    Cheers,

  • Dominic
    • The Incredible Code Injector

    Hi Predrag,

    Sorry for the delayed response, I don’t think I was notified that you’d responded.

    I’m using Google Chrome on a Windows 7 machine mostly, but have also tried in IE and Firefox on Windows 8.1, 10 and iOS on an iPad 3 and I get the same issue every time even when attempting to run the snapshot you created, which worked for you.

    I am well and truly stumped.

    Any more ideas?

  • Predrag Dubajic
    • Support

    Hi Dominic,

    I tried creating the snapshot one more time, and even asked one of my colleagues to do it as well since he is on Win 7 using Chrome and both times snapshots were created without any issues.

    Can you tell me if you are still seeing the same errors as described in your original post or you are seeing something differently now?

    Best regards,

    Predrag

  • Dominic
    • The Incredible Code Injector

    Hi,

    I’m running these sites on a VPS so don’t get much in the way of support here.

    Funnily enough, I’ve also got an issue with M2P memberships not expiring and producing the very same errors in my logs:

    Membership 2 Pro Memberships Not Expiring

    Don’t want to run 2 conversations on the same issue – looks like there’s something causing an issue for cron generally? Doesn’t explain why you are able to run these and I am not, but must be connected?

    Looking in my cron for this site, here are the jobs that are scheduled:

    itsec_purge_lockouts

    itsec_purge_logs

    ms_cron_check_membership_status

    ms_cron_process_communications

    pilates_pt_auto_mailer_hook

    redux_tracking

    w3_minify_cleanup

    w3_pgcache_cleanup

    w3_pgcache_prime

    wp_cache_gc

    wp_scheduled_auto_draft_delete

    wp_scheduled_delete

    I know what all these are related to – obviously W3TC, M2P, iThemes Security, Redux. “pilates_pt_auto_mailer_hook” is one from a custom plugin I coded on this site. Functions hooked to this are definitely working ok. I’ve also tried removing and re-adding this and am still seeing the same issues.

    Does this generate any further ideas??

  • Dominic
    • The Incredible Code Injector

    Hoang,

    Thanks for that, I have made the changes to the fcgid.conf file, and that appears to have resolved the symptoms! I have been able to observe 2 memberships expiring on time, so thanks so much for that.

    I have two further questions here though:

    1. While I’m pleased that this appears to have resolve the symptoms of this issue, I’m a bit concerns that I still don’t fully understand the cause. I think this has shown that wp-cron is frequently taking somewhere between 45s – 500s to complete, which I don’t think it should. Are you able to offer any advice on zeroing in on what processes or functions could be taking such a long time? Additionally, I’m sure that this timeout value was set to 45s for a reason, I’m nervous that allowing so much additional time could have an adverse affect on my server that I don’t fully understand yet.

    2. Separate point – I while observing the expiring memberships, I couldn’t help but notice that the automated email that is sent out by M2P was sent almost 24hours ahead of the membership being moved to expired status – this seems odd, do you know if this is expected?

    Thanks,

    Dominic,

  • Predrag Dubajic
    • Support

    Hi Dominic,

    I’ll need to ping @hoang Ngo again here for further explanation on the first issue.

    As for the second issue with M2 email, you can enable email on expiration and email X days before activation, are you sure it wasn’t the second one that is being sent?

    Could you open new thread regarding this other question so we can deal with it separately?

    You can start a new thread here:

    https://premium.wpmudev.org/forums/#question

    Best regards,

    Predrag

  • Dominic
    • The Incredible Code Injector

    Thanks Pradag, sorry, got my M2P / Snapshot threads mixed up there – shall I shut down the M2P thread?

    I have now also attempted a Snapshot run with the same snapshot that I had issues with previously and am happy to say that this has completed successfully generating no errors in my server logs.

    First question still stands here so hopefully Hoang can offer some ideas here.

    *EDIT* – Have just seen that my server reported an Apache memory alarm status change from Green > Red while I was running the Snapshot process.

    I have done some research, and am wondering if I could do something like –

    – create 2 functions: 1 to start a timer, one to stop

    – hook them to each cron hook currently registered on the site

    – record in the error logs the start and end time for each hook

    Will give that a go and report back.

    As for the M2P email issue, I’ll monitor a little longer and open a new thread if necessary as you suggested.

    Cheers,

    Dominic,

  • Predrag Dubajic
    • Support

    Hi Dominic,

    Thanks Pradag, sorry, got my M2P / Snapshot threads mixed up there – shall I shut down the M2P thread?

    No problem at all, it’s just easier to keep an eye on everything when there are separate threads to issues that are not closely related.

    Do you already have thread opened for that M2 issue?

    I’ll have to leave the second part of your question for Hoang as he is more code savvy than me :slight_smile:

    Best regards,

    Predrag

  • Dominic
    • The Incredible Code Injector

    Hey,

    Sorry, what I meant was that I had an M2P thread open for memberships not expiring, I’ll close that down because I think this is the same issue relating to Snapshot.

    OK, update from me, and hopefully useful to Hoang when he gets back to this, I mad a simple plugin to measure the time from when a cron hook is first fired, to when it ends. Using the R_Debug plugin ( https://gist.github.com/Rarst/1739714 ) I saw that there is only 1 function attached to any of these hooks, so my theory is that my timings should in fact measure the execution time of the function attached to the hook…

    I’ve pasted my code at the end of this post to show what I have done, it’s a bit of a hack but I just wanted to add something quickly.

    I’ve attached my results table after letting it run for 24 hours, plus a manual run of my core site’s snapshot. The values shown are the longest time taken by any run of each hook, i.e. some of these hooks may have been run multiple times, but I’ve only displayed the longest time taken of those. As you can see, it’s Snapshot and M2P that are taking a long time to run…

    Not sure where to go next to figure out how to resolve this!

    Here’s my code for adding the timer:

    $cron = get_option( 'cron' );

    foreach( $cron as $cron_entry ) {
    $x = 0;
    if( is_array( $cron_entry ) ) {
    foreach( $cron_entry as $key => $cron_name ) {
    if( $x == 0 ) {
    add_action( $key , 'bd_start_timer', 1 );
    add_action( $key , 'bd_end_timer' , 1000 );
    $x++;
    }
    if( x != 0 ) { break; }
    }
    }
    }

    function bd_start_timer() {

    $_SESSION['bd-timer-start'] = microtime_float();

    }

    function bd_end_timer() {

    $time = microtime_float() - $_SESSION['bd-timer-start'];
    error_log( 'Cron timer for ' . get_bloginfo( $show ) . ': ' . current_filter() . ': Total time: ' . $time );

    }

    function microtime_float() {
    list( $usec , $sec ) = explode( " " , microtime() );
    return ( ( float )$usec + ( float )$sec );
    }

    Cheers,

  • Hoang Ngo
    • Code Slayer

    @dominic,

    Usually, not much process will take more than 30s to completed. However, backup & restore, will require some minutes to be done. And at the stage I told you to update the FcgidIOTimeout to 500, I just want to test to see if that’s the reason for this issue.

    And the result is yes, and from your table (wow!!!!), it seems we only need 250s to finish. So now you can set the FcgidIOTimeout at 250 or better 300. In case your site will increase the size of files and data, and we surely don’t want the backup just stopped at sometime because this issue.

    If you know what are running on your server, all your plugins/themes installed from a known source, so I don’t think the FcgidIOTimeout will harmful.

    Please check this too http://stackoverflow.com/questions/3878700/is-there-a-way-to-have-apache-log-slow-requests. If you want, you can setup a log for slow requests.

    If you have any additional issues, please let us know and we’ll be happy to help.

    Best regards,

    Hoang

  • Dominic
    • The Incredible Code Injector

    Many thanks Hoang, I will check out the link you provided, I assume this will allow me to monitor for anything else that starts running for a long time and causing an issue?

    Anyway thanks to Predrag as well for sticking with this one as well, think we can consider this closed!

    Cheers,

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.