Snapshot stopped working properly

I logged in to the dashboard (network) and snapshots which were running weekly/monthly/daily stopped working. Managed backups stopped working properly since April as well.
I can click and run backups in the dashboard and download them to my computer just fine. But they won't run automatically and even the ones I create manually won't appear in Dropbox like they should.

  • Konstantinos Xenos

    Hi MechKW !

    Since there are multiple issues as it seems on your post let's take it one by one:

    Managed Backups:
    I've created a Managed Backup without any issues, all debug logs where clear and everything worked perfectly fine. You'll find it under the Managed Backups tab with today's date.

    Running Automatically:
    WordPress has WP_Cron to manage and process scheduled jobs but it's not the most on-time solution nor it always works to be honest. WP_Cron requires frequent users to be visiting your website so it's internal clock is updated for scheduled jobs to run on time properly. If a clock misses it's turn it might end up either missing the whole cycle or even multiple jobs trying to run resulting in an overload of resources so many things can be failing.

    The best solution is to disable WP_Cron and actually use your servers to always make sure that the scheduled jobs are running on time. More information can be found on the wp.org dev docs at https://developer.wordpress.org/plugins/cron/hooking-into-the-system-task-scheduler/ .

    Dropbox:
    To be able to debug that I'll have to request your permission to use my testing DropBox account instead, so I'll have to remove yours for the time being until the tests are done. Tell me if that's an option please.

    ---

    Extra Note: I see quite a lot of log files within our snapshot folder so I'll make sure to go through them as well in case I can pinpoint of why you where having issues since April as you say, but this process might take a while, especially since the Managed Backup worked fine at the moment it might have been any kind of updates plugin/server/wp that could affect anything as you can understand.

    Regards,
    Konstantinos

  • MechKW

    Ok cool. I'll wait and see what happens. In the mean time. I'll look into the system task scheduler suggestion. This makes me nervous though and I don't want to replace automatic cron job management with manual management by me that's going to need to be updated constantly.

    One other question. What is the process by which "Automate" initiates a backup? Is it WP Cron or external? Are all managed backups reliant on wp cron or are they externally initiated.
    I get messages like these "Aug 24, 2018 at 4:33am: Error when attempting full site backup" until Automate eventually gives up tyring to back up the site and then I re-enabled it.

    Thanks for looking into this.

  • Konstantinos Xenos

    This makes me nervous though and I don't want to replace automatic cron job management with manual management by me that's going to need to be updated constantly.

    A server side cron doesn't need an update basically. It's the other way around. A WP_Cron is what needs 'constant update' to read the clock and that's why it requires visitors. A server cron will actually 'check your website every 5 minutes' or at any time you will point it to run. So that acts as an 'automated visitor' let's say and the internal clock will always be updated and correct.

    One other question. What is the process by which "Automate" initiates a backup? Is it WP Cron or external? Are all managed backups reliant on wp cron or are they externally initiated.
    I get messages like these "Aug 24, 2018 at 4:33am: Error when attempting full site backup" until Automate eventually gives up tyring to back up the site and then I re-enabled it.

    On a minified explanation of https://premium.wpmudev.org/docs/getting-started/hub-automate/ . Automate will demand a backup ( this is an extra backup let's say and has nothing to do with any other backups that you have set via Snapshot ) before any updates and there are 2 processes after that. If the backup fails it will retry, but on 6 consecutive fails it will disable itself completely.

    ---

    I will log my dropbox account and see what's going on on that part as well and get back to you with a full rundown as soon as possible.

    Regards,
    Konstantinos

  • Konstantinos Xenos

    Hi again MechKW I'm back with some really interesting findings.

    First of let me say that I've added our account as an Administrator to virtue.studio as I needed access to a 'Tools' menu so I could check all the cronjobs.

    Dropbox:
    I've checked your dropbox account and even though it seems 'connected' it might as well have lost the Authorization so could you "Force Authorize" once more just to be sure on that part?

    I've added one of my own accounts and tested with a Snapshot as well and it didn't upload as you mention -> this lead me to the next part so read carefully please.

    --

    Now this is the interesting part. All of the cron-jobs are In queue and most / if not all without even showing when they are supposed to run. This means that they might as well be in queue indefinitely. Also I tried manually starting a snapshot cron-job without success. This means that something is either breaking crons or for some reason the access is limited as there are no errors returning a s well.

    Now I see in your debug.log some "<unknown file>:<unknown line> " throwing errors that I can't be sure of course from where they are originating but it's surely unusual for a php log to not being able to identify which file throws the error. There are also plenty of Divi related errors.

    One extra notice is that you're running some plugins that are monitoring everything ( i.e. activity monitor etc that might be halting crons or doing all kinds of tricky stuff behind the scenes ).

    Unfortunately this leaves me no choice to request for a theme / plugin compatibility test. This basically means that you'll have to disable all plugins and leave only Snapshot running to test, as well as reverting to a bundled theme ( TwentySeventeen for example ).

    This will narrow down the errors as it will be a 'plain' WordPress installation with no other plugin interfering.

    This is not an easy task by any means and it's always preferable to be done on a staging/dev environment and not on your live if that's a possibility.

    To sum it up. Is it possible to create a staging environment for me and I can do all the tests needed myself for you as I'll be working freely without fearing of breaking your live website.

    If not I'd need server-side Access to create full backups via the server myself and then proceed on checking anything on the live site ( of course it would be best to keep your own server-side full backups as well ).

    You can send me the information needed to make the changes for you privately through our contact form: https://premium.wpmudev.org/contact/#i-have-a-different-question by following this example:

    Subject: "Attn: Konstantinos Xenos"
    
    - Staging Admin login ( if Multisite please provide Super Admin details ):
    Admin Username:
    Admin Password:
    Login URL: 
    
    - Staging FTP credentials
    Hostname:
    Username:
    Password:
    Port:
    Key-File ( and password ) if needed
    
    - Server Admin ( CPanel / Plesk )
    Username:
    Password:
    Login URL:
    
    - Link back to this thread for reference
    - Any other relevant URLs -or- information regarding the issue that was not included in this thread

    Regards,
    Konstantinos

  • MechKW

    Hey I know this is old and I've been still looking for ways to make this better. I did the contab fix as suggested and I read that the CLI command (as opposed to the standard call) uses a separate process and has no time limits like standard php. So I though I'd run a cron --all directly from the terminal to see what happens.

    Results in full below but for now I'll summarize. My dropbox account started getting flooded with new files :slight_smile: and all the WEEKLY database only snapshots got updated to today (interesting but I'll take it). Here's the interesting part. The upload (as I read it, see below) took 17 minutes to complete, maybe just because it had so much catching up to do. But another thing the backup job (one of many) took over 5 minutes to complete. Would these time frames prevent normal automatic operation. Let me know what you think of this test.

    Also I'm concerned about using --allow-root as it warns that wordpress would have control of the entire system. I didn't mind this for a test but would not want that to be a permanent part of the solution.

    root@virtue:disappointed:[[REDACTED full path to wp dir]]# wp cron event run --all --allow-root
    PHP Notice: Constant WP_ALLOW_MULTISITE already defined in phar:///usr/bin/wp/php/WP_C LI/Runner.php(1138) : eval()'d code on line 52
    Executed the cron event 'wphb_performance_scan' in 3.595s.
    Executed the cron event 'wp_privacy_delete_old_export_files' in 0.021s.
    Executed the cron event 'cleanUpOldLog' in 0.014s.
    Executed the cron event 'psts_process_stats' in 0.025s.
    Executed the cron event 'wp_stream_auto_purge' in 0.005s.
    Notice: Undefined property: stdClass::$theme in /[[REDACTED full wp path]]/wp-admin/ includes/class-wp-automatic-updater.php on line 269
    Executed the cron event 'wp_version_check' in 0.722s.
    Executed the cron event 'wp_update_plugins' in 0.007s.
    Executed the cron event 'wp_update_themes' in 0.005s.
    Executed the cron event 'delete_expired_transients' in 0.027s.
    Executed the cron event 'update_network_counts' in 0.021s.
    Executed the cron event 'puc_cron_check_updates-admin-menu-editor-pro' in 0.146s.
    Executed the cron event 'wpmudev_scheduled_jobs' in 0.889s.
    Executed the cron event 'auditReportCron' in 0.006s.
    Executed the cron event 'lockoutReportCron' in 0.005s.
    Executed the cron event 'wp_scheduled_auto_draft_delete' in 0.123s.
    Executed the cron event 'et_builder_fonts_cron' in 0.017s.
    Executed the cron event 'wp_scheduled_delete' in 0.007s.
    Executed the cron event 'gravityforms_cron' in 0.007s.
    Executed the cron event 'upgrader_scheduled_cleanup' in 0.013s.
    Executed the cron event 'et_core_page_resource_auto_clear' in 0.021s.
    Executed the cron event 'upgrader_scheduled_cleanup' in 0.031s.
    Executed the cron event 'upgrader_scheduled_cleanup' in 0.003s.
    Executed the cron event 'upgrader_scheduled_cleanup' in 0.033s.
    Executed the cron event 'upgrader_scheduled_cleanup' in 0.003s.
    Executed the cron event 'defenderSubmitStats' in 1.373s.
    Executed the cron event 'upgrader_scheduled_cleanup' in 0.004s.
    Executed the cron event 'wphb_clear_logs' in 0.596s.
    Executed the cron event 'upgrader_scheduled_cleanup' in 0.003s.
    Executed the cron event 'snapshot_remote_file_cron' in 1042.297s.
    Executed the cron event 'snapshot-controller-full-cron-rotate_local_backups' in 0.042s.
    Executed the cron event 'snapshot_backup_cron' in 6.366s.
    Executed the cron event 'snapshot_backup_cron' in 3.71s.
    Executed the cron event 'snapshot_backup_cron' in 6.914s.
    Executed the cron event 'snapshot_backup_cron' in 2.831s.
    Executed the cron event 'snapshot_backup_cron' in 319.396s.
    Executed the cron event 'snapshot_backup_cron' in 4.35s.
    Executed the cron event 'snapshot_backup_cron' in 4.328s.
    ...did not return to prompt (still going?)

  • Leonidas

    Hi there MechKW ,

    looking at the log file you shared, I have to report the following:
    There are two jobs in there that are significantly longer than the others:
    snapshot_remote_file_cron - 1042.297s and
    snapshot_backup_cron - 319.396s

    Looking at your Snapshot configuration, both these jobs make sense. snapshot_remote_file_cron , which was the longest, clocking at 17 minutes, is the job responsible for forwarding backups to the remote destinations. Since this job actually handles all pending backups to be uploaded, it is natural to take that long, especially since there are plenty of recurring snapshots at your site. This can be reduced if running cron jobs via the WP-CLI is configured to run in a frequent enough interval. That way, each time snapshot_remote_file_cron is fired off, the upload queue is emptied and the next time the job fires, it will be responsible for uploading only the backups created between those two jobs.

    About the snapshot_backup_cron that takes about 5 minutes to complete:
    this should be the job responsible for taking the backup of the main site which is aprox. 1.2 GB. Again, 5 minutes is a sensible period for a backup that big and isn't likely to intervene with the proper use of crons at your site. However, you can set it up to run at its own time slot via the snapshot's settings, just to prevent excessive resource usage for multiple simultaneous snapshots.

    Taking a look in your site, I noticed that the dropbox snapshots in there have been uploaded to their destination, so as long as the WP-cron doesn't work properly on your end, the WP-CLI crons are the way to go for your scheduled remote snapshots to work. In the event you want to not rely on WP-CLI for running the cron jobs, there should be a theme/plugin conflict test, much like my colleague Konstantinos proposed, in order to figure out whether the WP-cron is blocked by one of the themes/plugins.

    Best regards,
    Leonidas

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.