snapshot timeout error - databases backup but media times out

all my databases are backing up just fine, but when it gets to media, zero progress is made and it eventually times out with the following error:

Request Timeout
This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.

this is a litespeed server, and i've added the following to .htaccess
RewriteRule ^$ – [E=noabort:1]

I've tried backing up to dropbox initially, but also tried locally... both end in the same timeout.

any help is appreciated

    Tyler Postle

    Hey lloyd,

    Hope you're doing well today!

    What is your max execution time? You can check this via:

    WPMU DEV > Support > System Info. It will be the 2nd number from the top. You might want to try increasing that even though it shouldn't be aborting - just to be sure. Add this to your php.ini file:

    max_execution_time = 60

    If it's still not working then let us know and we can have another look

    All the best,
    Tyler

    lloyd

    Hi Tyler,

    Thanks for getting back to me so quickly.

    max_execution_time is 60 (as it should be?)

    There is quite a few pictures for this particular site, as it is an e-commerce site (running woo)

    i've used backupbuddy previously and the total backup was around 400MB (not sure if this gives you a better idea of what i'm dealing with) however i'm not sure this is considered large/alot.

    thoughts?

    Tyler Postle

    Hey Lloyd,

    No problem, happy to help.

    400mb is quite a large backup, I'm guessing most of that would be your media. In that case you might need higher than 60 seconds. Maybe try 300 instead, which would be 5 minutes. You can always lower it after too

    Should give us an idea of whether that is actually the issue, I'm thinking it is.

    Look forward to hearing back.

    All the best,
    Tyler

    Jeff

    Hi Tyler, if this should be a new separate post please move it if possible. I am having the same issues described here with the timeout.

    I have Snapshot installed in a number of sites, of various sizes, and they all have this happen if I include any files. I have adjusted the max_execution_time to 600, and it still happens.

    I thought maybe it was due to the size of the backup (as you noted 400mb is quite large) and have tried leaving out uploads/media, it failed. I tried leaving out plugins as well, it failed. I have a site with just 2 measly themes and it failed.

    It was able to complete with just the config & htaccess files backed up, but no other file inclusions seem to work. Do you have any ideas? Could you take a look at logs and see if something on the host is to blame?

    Tyler Postle

    Hey Jonathan,

    It is better to create a new thread rather than post on a really old one as that way it gets added and assigned to an active staff properly. Sorry about the delay! Since I'm here now though we can just continue here so that is for future reference.

    Do you have any ideas? Could you take a look at logs and see if something on the host is to blame?

    Can you grant support access? Then I can have a look at the logs

    You can grant support access via WPMU DEV > Support > Grant Support Access.

    http://premium.wpmudev.org/manuals/wpmu-dev-dashboard-enabling-staff-login/

    Also, have you tried the Managed Backups? Or was this with the Managed Backups?

    You can find those via:

    Snapshot > Managed Backups.

    The reason I ask is that the Managed Backups backup your site in a completely different method than the regular snapshots. If you get a timeout or memory error with regular Snapshots there is a good chance that Managed Backups may still work as it breaks the backup process into much smaller chunks.

    Talk to you soon!

    Cheers,
    Tyler

    Jeff

    Thank you Tyler, I have granted support access to one of the sites we manage.

    I had not tried Managed Backups, I see that's a new option in beta. I tried it on another site (not the one granted access) and it gave an error:

    Request Timeout
    
    This request takes too long to process, it is timed out by the server. If it should not be timed out, please contact administrator of this web site to increase 'Connection Timeout'.

    Let me know what you find. Thanks!

    Tyler Postle

    Hey Jonathan,

    Apologies for the delay and I should have mentioned this in my last post, I planned to use the Managed Backups in-dash error reporting features. Can you enable Managed Backups again? It's error logging should give us an idea of what is causing this you will need to activate it with the unique key found in your dashboard here: https://premium.wpmudev.org/hub/my-websites/?site_id=508255&tab_slug=backups#key

    Sending in your FTP would be helpful as well and likely avoid us having to ask for it later on. Regular snapshots save their logs in the _logs folder via: /wp-content/uploads/snapshots/

    That should also give us some additional info.

    You can send that privately through our contact form: https://premium.wpmudev.org/contact/

    Select "I have a different question" for your topic - this and the subject line ensure that it gets assigned to me.

    Send in:

    Subject: "Attn: Tyler Postle"
    -WordPress admin username
    -WordPress admin password
    -login url
    -FTP credentials (host/username/password)
    -link back to this thread for reference
    -any other relevant urls

    **If you keep support access active then no need to send in wp-admin

    Talk to you soon! Cheers

    Tyler Postle

    Hey Jonathan,

    It looks like the Managed Backups worked properly for your redmark site. It backed everything up fairly quickly. I can see that using the older originally Snapshot method that it gets stuck on the /media/2015/06 folder. That appears to be where it times out.

    The reason is doesn't with the Managed Backups is that managed backups breaks everything into equal size chunks instead of trying to backup whole folders at a time.

    Your other site that fails on the managed backs ups as well, we could reduce the chunk size, but it might be better to try increasing the execution time first. Is the max execution time for that site 600 as well? If so, then try adding this to your wp-config.php to reduce the file chunk size:

    define('SNAPSHOT_FILESET_CHUNK_SIZE', 25);

    Look forward to hearing back!

    Cheers,
    Tyler

    Jeff

    Tyler, this change to wp-config.php seemed to do the trick, even for local backups. I've tested it on a couple sites and had success. Thank you for continuing to investigate.

    I've found another problematic site, though. It may be due to multi-site setup, I'm not sure. The backup looks like it's working just fine and then gets stuck again at the files portion. It also failed with the same error on the managed backup option. This is with the chunk size setting in wp_config.php.

    I will share admin access via the site if you can take a look. Thank you!

    Tyler Postle

    Hey Jonathan,

    Sorry for the late reply! Glad that has been working for the most part.

    As for the problem site. Yeah I see it got a connection timeout error when I just tried a managed backup.

    Can you try making that chunk size even a bit smaller on that site? ie.

    define('SNAPSHOT_FILESET_CHUNK_SIZE', 20);

    It does look like it stopped on files again; however, perhaps it had just gotten to the database and then timed out on the first database chunk, so you can try making the database table chunk size smaller too:

    define('SNAPSHOT_TABLESET_CHUNK_SIZE', 100);

    This will result in several more requests; however, doesn't look like your site is having any trouble with the number of requests it's just the size of them.

    I deactivated your scheduled backup as you will want to add/change those values with automatic backups turned off. Then turn it back on after, although easiest to just run a manual Managed Backup after you have set those constants.

    Keep in mind, this won't affect the regular Snapshot backup at all, only the Managed Backups. The regular backups don't use the chunk method and will try to do everything at once, which on shared hosting can be problematic or if your site is very large - so in your case it would be best to stick with the Managed Backup method.

    Let me know how it goes

    Cheers,
    Tyler

    Jeff

    Tyler, it still failed after reducing the chunk size and adding the table chunk size limit too. I've renewed support access if you are able to look again and troubleshoot.

    If there's anything else we can do to identify a solution to this let me know. Additionally, I would love to have a fix for sites that don't use the managed backup method. I just need a reliable and painless backup solution.

    Tyler Postle

    Would you be able to send us your cPanel so we can check your server setup and error logs there? That will be the next step to troubleshooting this. Once we have that I'll also ask our Snapshot developer if they can have a look to get their valuable feedback

    You can send that privately through our contact form: https://premium.wpmudev.org/contact/

    Select "I have a different question" for your topic - this and the subject line ensure that it gets assigned to me

    Send in:

    Subject: "Attn: Tyler Postle"
    -cPanel login url
    -cPanel username
    -cPanel password
    -link back to this thread for reference
    -any other relevant urls

    **If you keep support access active then no need to send in wp-admin

    As for the sites that don't use the Managed Backup method and are getting timeouts with the regular Snapshots, unfortunately there is not much we can do there as it just comes down to the fact that your server can't handle the backup via PHP application. In those cases, the best option is to use a server side backup solution, so something like the cPanel backup options. OR you could increase your timeout further.

    With that said, once we have cPanel access we can double check if there is anything we can do server side to help with regular Snapshots too.

    Look forward to hearing back!

    Cheers,
    Tyler

    Tyler Postle

    Hey Jonathan, I actually haven't received an email yet, I checked our spam box as well and don't see anything from you there either.

    Sorry about that, we did have an issue with our email for a short period last night, not sure if that is when you sent it.

    Either way, can you re-send it directly to contact@wpmudev.org instead of using the form? That should work you can reply back here again after it's sent just in-case.

    Still make the subject "Attn: Tyler Postle"

    Talk to you soon! Cheers

    Tyler Postle

    Hey Jonathan,

    Thanks for sending that in. I got them this time

    I've been debugging this further and the issue was that you had a very large 500+mb zip file in the root directory. So when it included that as part of a chunk, it was still a huge chunk size wise and timed out.

    I moved the "files.zip" folder into the /tcole/ directory - so it wouldn't be included in the Snapshot. After I ran the Managed Backup again, it did make it further but then I got an "empty response" error. I've turned on debug logging, so it will create a debug.log file in your /wp-content/ folder. It takes a moment for that to kick in.

    Can you try running a Managed Backup now and then let me know when it's done and I'll check the debug.log. Since it takes a little bit for the new debug.log to be generated I can't run it now.

    I do want to say, that even if we get this working now - as your site grows and you add bigger files to it again, you will likely run into the same issue. I really would recommend going with a server side managed backup in your case since your site is almost 4GB.

    The only other option would be to keep increasing your timeout limit.

    Look forward to hearing back!

    Cheers,
    Tyler

    Jeff

    Thanks Tyler. I have initiated a managed backup and it eventually returned the "aw shucks" error with "<empty response>" so when you're able to, please look at the debug.log.

    The "files.zip" archive was there as a result of manually backing up the files. I guess it had been left there after creating & downloading. Sorry about that. I'll remove it entirely as well as a few other archives I spotted, and retry another managed backup. Also, it's possible that this particular site is larger as there's essentially two copies of it (live, dev/staging) via Multisite.

    I understand the size concerns. If the server storage isn't an issue and the individual files are not massive, it shouldn't matter how many files (i.e. uploaded jpg, pdf files) are backed up? Or does the archive created become a problem?

    We do have scheduled backups on the server, but looking for a manual solution to easily create backups prior to applying theme, plugin, and WP core updates. Maybe snapshot is not ideal for this, but that wasn't obvious when we set out to use it this way.

    Tyler Postle

    Thanks for running that - doesn't look like the debug.log was created; however, I do see it's showing the notices/errors in your error_log. With that said, none of them look to be related to Snapshot or the backup process.

    I'm going to ask our developer if they have any other ideas what might be causing that.

    I understand the size concerns. If the server storage isn't an issue and the individual files are not massive, it shouldn't matter how many files (i.e. uploaded jpg, pdf files) are backed up? Or does the archive created become a problem?

    In your case the archive doesn't become a problem because you have your Snapshots saved outside of your WP root install.

    That is correct that if the files aren't massive and server storage/bandwidth isn't an issue then it shouldn't matter how many files there are as it will continue backing up the chunks until it is finished; however, when a process is running for an extended period of time it can be a red flag for a server and it might terminate it as it could affect performance.

    When your site starts getting several GB's in size, it's not usually a good idea to use a PHP application(any plugin) for the backup, due to all the limits imposed by the server. So I don't have any other plugin recommendations, but I have asked the developer about this as well - see if they have any recommendations

    Once I hear back from the developer on both of the above queries I'll reply back here.

    Hope you have a great weekend, talk to you soon. Cheers

      Tyler Postle

      Apologies for not getting back earlier on this! I spoke with the developer and they basically said the same thing as above. That once your site starts getting quite big then a PHP based application really isn't the best option and you should be backing up from the server side.

      cPanel comes with it's own server side backup tool so that could be an option for you. May not be quite a user friendly as a plugin would be, but it should definitely work for even larger sites and would be compatible with any other cPanel based hosting if you were to move your site.

      Hope that helps!

      Cheers,
      Tyler