Snapshot is not completing backups

Hi,

We are trying to create a backup of a multisite network using Snapshot. It is a network of two sites with a relatively large database (over 700mb) and uploads folder (about 30Gb).

We tried creating snapshots both scheduled or immediate, including common files, all database tables and saving on a Dropbox account (keeping local archives as well).

In the end, we are getting the snapshots folder of a random size (620Mb, 1.3Gb etc) on the local server. It created a folder on Dropbox but it is empty. The snapshot is visible in the admin panel but if we click restore, it is empty. The log looks as follows:

[Remote][2017-03-03 08:49:33][Warning] API connection to get-token returned non-200: 401
[Remote][2017-03-03 08:49:33][Warning] Unable to get fresh token
[Remote][2017-03-03 08:49:34][Warning] API connection to register-settings returned non-200: 401

How do we create a complete backup?

    Dimitris

    Hey there Brian Brady,

    hope you're doing good and thanks for reaching us!

    We could try raise up some server limits to check if that makes any difference (and in case that you haven't done that already).

    Could you please access your server through FTP, edit the wp-config.php file and insert the following line just above the /* That's all, stop editing! Happy blogging. */ comment

    define('WP_MEMORY_LIMIT', '512M');
    define('WP_MAX_MEMORY_LIMIT', '512M');

    Ref: https://codex.wordpress.org/Editing_wp-config.php#Increasing_memory_allocated_to_PHP

    Then you can try to raise

    max_execution_time = 300
    default_socket_timeout = 300

    as described here https://premium.wpmudev.org/blog/increase-memory-limit/
    (you may have to contact your hosting provider for this)

    You should then be able to see these changes in WPMUDEV -> Support -> System Info [button] under the PHP and WordPress tabs.

    Also, I'd like to enable some debug information that might help us narrow down this. Access your server via FTP, edit the wp-config.php file, find a line like
    define('WP_DEBUG', false);and replace it with the following (if the above line doesn’t exist, simply insert next snippet just above the /* That's all, stop editing! Happy blogging. */ comment)

    // Enable WP_DEBUG mode
    define('WP_DEBUG', true);
    // Enable Debug logging to the /wp-content/debug.log file
    define('WP_DEBUG_LOG', true);
    // Disable display of errors and warnings
    define('WP_DEBUG_DISPLAY', false);
    @ini_set('display_errors', 0);

    Then go ahead and try to perform another snapshot locally. By doing so, a /wp-content/debug.log file should be created. Simply download it, rename it to debug.txt and attach it here in your next reply. If file size exceeds the 5MB limit of our forums, please use a service like Dropbox to upload it and post the shareable link in your next reply.

    Finally, grant us with support access to your website via WPMUDEV Dashboard plugin as described here: https://premium.wpmudev.org/docs/getting-started/getting-support/#chapter-4

    Warm regards,
    Dimitris

    Dimitris

    Hey there Brian Brady,

    hope you're doing good and please excuse our late reply here, it's been quite busy in our forums lately and we try to keep up with every member here!

    I review the debug log file you've kindly shared and there's no relevant information here, apart from a PHP parse error coming from your child theme which should be resolved pretty soon as it got logged only once.

    Could you please grant us with support access to your website via WPMUDEV Dashboard plugin?
    https://premium.wpmudev.org/docs/getting-started/getting-support/#chapter-4
    Let us know here in your next reply that access has been granted as we don't get any notifications about it.

    Warm regards,
    Dimitris

    Predrag Dubajic

    Hi Brian,

    Thanks for granting access, I did some tests on your site and I was able to create database and file backup (without media) but the Media backup alone fails and gives me 504 Gateway Time-out error.
    504 error is usually server related and since you have large media folder it's possible that your server can't handle high number of requests and because of that it fails.

    Can you ask your hosting provider to check server logs and see if we can get some more info from there?

    Best regards,
    Predrag

    Predrag Dubajic

    Hi Brian,

    Snapshot on its own doesn't have a limit to maximum size of folder/file that it can handle, however it can be limited by server itself, it does require more resources in order to create backup of large folders and if server can't handle the number of requests it can fail like this.

    You could try creating multiple backup files in order to split your media folder in smaller sizes.
    When you add/edit snapshot you will see exclusion field, in there you can exclude specific folders from your backups.
    This way you can exclude all but one media folders and split them in multiple backups so it takes less resources to complete and hopefully that would work.

    Best regards,
    Predrag

    Brian Brady

    Hi,

    Ok, sure. We have the following configuration: CentOS server with 11 CPU Cores, 11GB RAM, 140GB SSD and this is not enough to make a backup of the uploads (27Gb) folder alone. What are your hardware recommendations so that server can handle the number of requests to make a backup of the site described above? What are your PHP setup recommendations?

    Excluding separate folders from uploads won't work because all media files are in this folder and are not categorized by months.

    Adam Czajczyk

    Hello Brian!

    The hardware that you are using should be more than enough to handle that. I checked the site and PHP/WP versions and memory limits look good enough either.

    There are however some additional PHP settings that might be worth adjusting as they might be affecting the issue:

    1. "max_execution_time" is currently set to "0" which means "unlimited"; however, there might be other limits and in case of Apache web-server (which is used on your host) there's a "timeout" directive that's by default set to 300 seconds; that means that with "unlimited" max_execution_time setting scripts would probably be terminated after 300 seconds max.

    That said, I think it might be better to try to set "max_execution_time" to e.g. 600 or even 1200;

    2. "upload_max_filesize" is set to 2M only
    3. "post_max_size" is set to 8M only

    I'd suggest setting both these values to 128 or 256M and then test snapshot again.

    Can you also tell me please what sizes are some of the biggest single files in your media folder/Media Library?

    Best regards,
    Adam

    Brian Brady

    Hi,

    Accordingly your recommendations,
    max_execution_time is set to 1200
    upload_max_filesize and post_max_size are set to 256M

    But still no luck.

    Here's the top 30 largest folders/files in the uploads folder (sizes are in KiB):

    6961576 ./wpallimport
    6185568 ./sites
    6185520 ./sites/12
    4295976 ./wpallimport/files
    3593828 ./snapshots
    2921036 ./snapshots/_backup
    1803304 ./snapshots/_backup/1490172528
    1735248 ./wpallimport/uploads
    903976 ./snapshots/_backup/1490172528/snapshot-backup.zip
    899324 ./snapshots/_backup/1490172528/snapshot-backup.zip.dWjYLh
    616556 ./sites/12/wpallimport
    495124 ./wpallimport/uploads/f5090379f548f82c05430560d8aeb3dd
    470032 ./snapshots/_logs
    461652 ./snapshots/_logs/_sessions
    394664 ./snapshots/_backup/1489512335
    377824 ./wpallimport/logs
    363812 ./snapshots/_backup/1488895156
    362384 ./snapshots/_backup/1488895156/wp_postmeta.sql
    356256 ./sites/12/wpallimport/uploads
    312868 ./wpallimport/uploads/797660332a1fded57ba045631b7bb5b2
    312868 ./sites/12/wpallimport/uploads/797660332a1fded57ba045631b7bb5b2
    304096 ./wpallimport/temp
    268596 ./wpallimport/uploads/f5090379f548f82c05430560d8aeb3dd/Switch_Desks_v42.xml
    249780 ./sites/12/wpallimport/files
    248424 ./wpallimport/jpgs
    226528 ./wpallimport/uploads/5dfbfbfa3cbf0d5a6f3c9f8e584ba982
    226524 ./wpallimport/uploads/f5090379f548f82c05430560d8aeb3dd/Switch_Desks_v42.csv
    226524 ./wpallimport/uploads/5dfbfbfa3cbf0d5a6f3c9f8e584ba982/Switch_Desks_v42.csv
    226524 ./wpallimport/files/Switch_Desks_v42.csv

    Adam Czajczyk

    Hello Brian!

    Thank you for your replay.

    If I correctly read files' sizes, there are several files/folders over 1GB. Could you try temporarily exclude them from snapshots?

    You can do this on "Snapshot -> Settings" page in "Global Exclusions" section.

    If that helps, start removing the "smallest" ones until that breaks again and we should got some clue then. If that doesn't help, try adding next files/folder to exclusion list - those smaller than 1GB, one by one, starting from the "biggest" one.

    Hopefully, that will make it work at some point and will show us which file(s)/folder(s) are "standing in a way".

    Best regards,
    Adam

    Brian Brady

    Hi,

    I have excluded the folders that are above 1Mb (except the snapshots folder, which I assume should be excluded by default) but still ran into a timeout http://pix.toile-libre.org/upload/original/1491123246.png.

    I have a couple of questions regarding this.

    Why do we have "nginx" under this timeout message? We are running apache.

    The timeout is happening before 20 allowed mins. What can be reasons for that? Do you have any timeout reports specific for SiteGround servers?

    Regards,
    Igor

    Dimitris

    Hey there Brian Brady,

    hope you're doing good today!

    I have excluded the folders that are above 1Mb

    I assume that's a typo and you've excluded the ones over 1GB.

    Why do we have "nginx" under this timeout message? We are running apache.

    Because SiteGround's VPSs seem to be running both. Usually Nginx is in front of Apache as a reverse proxy.
    Ref: https://www.digitalocean.com/community/tutorials/apache-vs-nginx-practical-considerations#using-apache-and-nginx-together

    The timeout is happening before 20 allowed mins. What can be reasons for that? Do you have any timeout reports specific for SiteGround servers?

    I use a shared SiteGround server for testing and I'm able to use Snapshot (good old snapshots and managed backups) with no issues. I really can't tell why the limits we raised didn't make the trick here. Could you please send us over some server details so we could involve plugin's lead dev in your case?
    As this is public forum, you should send us your details through our safe contact form https://premium.wpmudev.org/contact/#i-have-a-different-question using this template:

    Subject: "Attn: Dimitris"
    - WordPress admin (login url/username/password)
    - FTP credentials (host/username/password)
    - cPanel/Plesk credentials for DB access (host/username/password)
    - link back to this thread for reference
    - any other relevant urls

    Keep in mind the subject line as ensures that it gets assigned to me.

    Take care,
    Dimitris

    Dimitris

    Hey there Brian Brady

    hope you're doing good and I appreciate the server credentials you've provided!

    As you've currently deactivated Snapshot, could you please re-activate it, insert your Managed Backup API key, set the debug level to "Info", for all stages, and before trying a manual backup, add the following snippet in wp-config.php file, just above the /* That's it! Stop editing... */ comment.

    define('SNAPSHOT_FILESET_CHUNK_SIZE', 10);
    define('SNAPSHOT_TABLESET_CHUNK_SIZE', 100);
    define('SNAPSHOT_FILESET_USE_PRECACHE', true);
    define('SNAPSHOT_FILESET_LARGE_FILE_SIZE', 104857600); //100MB

    Then hit the "Backup Now" button and leave the browser window open until finishes.
    Or you can schedule a daily backup instead and wait for it to finish.

    Let us know about your results!
    Warm regards,
    Dimitris