My website goes down when I'm trying to create a snapshot

I am trying to create a once off Snapshot, but my site goes down every time I try to do it and it doesn't complete the request.
I've tried to deactivate all plugins and switch the theme, but it doesn't help.
There are a lot of such waringins in the debug.log
PHP Warning: Error while sending QUERY packet. PID=51613 in ***/wp-includes/wp-db.php on line 1924

  • Huberson

    Hello Rogerio Barreto
    I've had a look at this and what I can tell from the logs is this issue might be caused by some limitation on the server and MySQL max_allowed_packet_size.
    Try to reach out to your hosting provider so they can assist you in increase the max_allowed_packet_size see if that takes care of the issue.

    There's also another error which apparently caused by MYSQL process being overused, that they might also want to look at in case it also affects the backup.
    mysqli_real_connect(): (HY000/1135): Can't create a new thread (errno 11 "Resource temporarily unavailable"); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
    You can find some details on that error here and a fix for it: (HY000/1135): Can't create a new thread..

    Do let us know if the above don't address it so we can further look into it.

    Regards,
    Huberson

  • Rogerio Barreto

    Thanks for the feedback Huberson. I did contact InMotion, my hosting provider, and although I do have a very robust shared hosting plan they told me that in this kind of plan they can not change the MySQL max_allowed_packet_size and advised me to hire a VPS with then.

    The higher tier of Support reviewed my situation and the tips you shared with me. Since they can not increase the MySQL max_allowed_packet_size on shared planes, they did optimize my database for this particular multisite installation website using phpMyAdmin, but although it did feel to me that the response was faster, it didn't solve my issues regarding setting up backups and importing demos from the theme I have purchased: Houzez.

    I am just wondering if a robust shared hosting plan would really exhaust its resources to deal with 4 multisite installation and 6 other regular WP installation. Since we don't use nor even 1/8 of the total storage limit available at the plan, I really couldn't tell if these amount of WP installation would be enough to exhaust the other resources of the plan.

    What would be your recommendation regarding my profile and the better hosting plan to deal with about 10 WP installation (4 multisites and 6 regulars). Would VPS be the most indicated?

    Regarding the error (HY000/1135): Can't create a new thread. I will read the details and the fix for it to see if this could help me out on that.

    Thanks a lot for the reply and support.

    Best regards,
    Rogerio

  • Adam Czajczyk

    Hi Rogerio,

    Thank you for your response!

    The "PHP Warning: Error while sending QUERY packet...." error is usually a result of the other one that's in your debug.log, actually. "Packet size" should rather be fine but the "error flow" here seems to be:

    Snapshot's performing some DB operations -> db gets "overloaded" (too many requests) -> db engine is not able to trigger new threads to handle all those requests -> request fails "on the go" resulting in packet error.

    In your previous post you're wondering whether decent shared hosting can be that "easily" exhausted by serving just 4 multisites and 6 regular single installs, while not even 1/8 of storage limit is used. The problem, however, is in quite a different type of resources. "Shared" environment means that CPU power, RAM but also - more importantly - webserver (let's say Apache) and db engine are also shared and not only among your 4+6 sites but with other sites (of other customers) on the same server as well.

    As a result, host must be limiting resources somehow to avoid cases where e.g. one badly optimized or even infected site takes down tens or even hundreds of sites of other users; or when a site of customer X eats up most of available resources, casing non-related sites of 100's of other customers to slow down to a drunken snail speed :slight_smile:

    So, resources that are usually limited (as most critical) are numbers of available webserver and database threads per account, as well as numbers of consecutive requests per e.g. hour/day and parallel requests.

    That leads me to an answer to your doubts: yes, even a really decent shared host can sometimes be not able to handle multiple sites. That depends on many different aspects and while those limits that I mentioned above are sort of "critical" here, things like how well your sites are optimized, how many requests they make (which also heavily depends on the theme and set of plugins that you're using) and even how big traffic they are getting :slight_smile:

    So, what could be done if host is not able to allow more db threads? That's unfortunately a bit "tricky" as we don't know exactly (and have no way of knowing) what and why exactly is "eating up" those limited resources. I mean, we do know that Snapshot backup is when it happens but whether it is due to Snapshot itself or is it a result - that's a different story here.

    That being said, I tried to check your site again and I noticed that even the wp-admin area is loading quite slow which might mean that the server indeed is overloaded. Basically, it's always trying to "queue" requests so if it does that successfully, site's working - might only be a bit slower; but if it cannot queue them all, something (worst case scenario: entire site) goes down.

    Unfortunately, I wasn't also able to use cPanel because the access credentials that you provided during the chat doesn't seem to work and I'm getting "wrong username" error with access denied. There's is a couple of things that I would like to check so could you please double check current cPanel access credentials and leave me a message in the text field under the green "Access active for..." button on "WPMU DEV -> Support" page in your site's back-end, containing working cPanel credentials?

    Let me know here after you do this and I'll check the install again.

    Kind regards,
    Adam

  • Rogerio Barreto

    I have notice that this very same issue is affecting other WPMU DEV members and I believe it goes a bit further so I will try to elaborate it here adapting the text below to the other issues I found out in the very same topic:

    1) When I create a Full Website Backup straight from my cPanel, the process is quite fast and does not consume that many server memory. It would be a good idea to suggest to the developers of the Snapshot PRO plugin to try to "optimize" the plugin work-ability, what I mean by that is to try to adjust the plugin to perform the very same service, but using way less server resources, otherwise it will be a plugin not very user friendly for those on a shared hosting environment.

    2) When clicking on Creat a Snapshot from the plugin panel, the process seems to exhaust server MySQL limits and due to that it stalls and doesn't complete the process. But for some reason, the schedule backups, although consuming a lot of server resources while proceeding a backup, goes through and complete the quest. So it seems to exist some kind of difference in the process of the schedule backups and the ones create on demand (that may consume even more servers resources).

    3) The time set up to run the backups is not working either. Since I have already notice that this is an extremely slow process and high resource usage, I have set them up to run late at night (when the site traffic would be its minimum) but the process is running in the afternoon.

    4) Although I have configured the Snapshots to send my backups to Google Drive, they are not being sent to my Google Drive Folder and they are sending the backups to my local server, what is not desired or I will very soon run out of Disk Memory. This is an issue that also seems to be affecting other members.

    5) Another thing that is not happening and also will contribute to increase the Disk Memory Usage is that besides the snapshots are not being sent to Google Drive, they are not overriding the oldest ones once reached the limit selected to be kept.

    6) Another trick issue is that several tests were performed in the last few days by me and by the support team and I was really surprise yesterday to find out that my sites had increased its memory usage from whopping 500MB to almost 4GB in the period. The reason for that is because every test not completed created a partial backup at the following folder public_html/wp-content/uploads/snapshots/_backup. I deleted them all yesterday.

    7) When I deleted a Snapshot from my WP Panel it didn't delete the respective file from my server. I am not sure if this was suppose to happen but I was expecting that once deleted from Snapshot Panel it also would delete the file from my server. So I deleted it manually from the server.

    So I believe there are some necessary developments for the next updates of the plugin to make it more reliable and lighter weight to be used.

    Best regards,
    Rogerio Barreto

  • Kasia Swiderska

    Hello Rogerio,

    Thank you for your feedback about Snapshot and new credentials - I'm checking with Adam what he wanted to check on cPanel and we will let you know about our findings.

    1) When I create a Full Website Backup straight from my cPanel, the process is quite fast and does not consume that many server memory.

    Server backups are very different than backups done via any plugin in WordPress. Server backups are done with tools like rsync and they operate on OS level. When you use plugin tool for backup it uses PHP to create backups - so on Apache level there is PHP running and there plugin works.
    On some servers with managed Backup you can use this define

    define('SNAPSHOT_ATTEMPT_SYSTEM_BACKUP', true);

    to use system binaries to do the backup. This should be generally much faster (or even exponentially decrease the time required for a managed backup) than normal managed backups via PHP, but requires: server support for executing binaries from PHP (PHP functions escapeshellarg, escapeshellcmd and exec are present and available), and presence of expected binaries needed to do the backup (zip, ln, rm, mysqdump and, optionally, find).

    2) When clicking on Creat a Snapshot from the plugin panel, the process seems to exhaust server MySQL limits and due to that it stalls and doesn't complete the process. But for some reason, the schedule backups, although consuming a lot of server resources while proceeding a backup, goes through and complete the quest.

    I'm checking this with our developers - because one difference is that scheduled Snapshots are triggered by WP Cron. I'll let you know when I will get feedback from developers.

    3) The time set up to run the backups is not working either. Since I have already notice that this is an extremely slow process and high resource usage, I have set them up to run late at night (when the site traffic would be its minimum) but the process is running in the afternoon.

    That is most probably because of WP Cron. By default scheduled Snapshots are triggered by it and WP Cron relays on traffic - so if there is no traffic on site it won't run and won't trigger Snapshot. Solution is to switch to Real Cron. This way it will work even if there is no traffic on the site.
    https://www.inmotionhosting.com/support/website/wordpress/disabling-the-wp-cronphp-in-wordpress

    4) Although I have configured the Snapshots to send my backups to Google Drive, they are not being sent to my Google Drive Folder and they are sending the backups to my local server, what is not desired or I will very soon run out of Disk Memory. This is an issue that also seems to be affecting other members.

    I checked your site and Google Drive throws error
    Google_0814_Exception: Failed to start the resumable upload

    that is most often related with incorrect configuration - please see here https://premium.wpmudev.org/forums/topic/manual-backup-error-while-running#post-1256665

    5) Another thing that is not happening and also will contribute to increase the Disk Memory Usage is that besides the snapshots are not being sent to Google Drive, they are not overriding the oldest ones once reached the limit selected to be kept.

    Can you enable again access? I was checking this issue on your site when it expired :slight_frown:

    6) Another trick issue is that several tests were performed in the last few days by me and by the support team and I was really surprise yesterday to find out that my sites had increased its memory usage from whopping 500MB to almost 4GB in the period. The reason for that is because every test not completed created a partial backup at the following folder public_html/wp-content/uploads/snapshots/_backup. I deleted them all yesterday.

    It looks like this is related to issue with Google Drive - if Snapshot would be send (so whole operation would be successful) they should be removed. Please check if this will be still happening with Google Drive destination working.

    7) When I deleted a Snapshot from my WP Panel it didn't delete the respective file from my server.

    I'm not sure if I understand correctly - you deleted plugin but it files remained on the server? Or you mean the Snapshots it created were still there?

    kind regards,
    Kasia

  • Rogerio Barreto

    Hi Kasia,

    For the solution to speed up the process using system binaries to do the backups unfortunately that won’t work since some of those requirements are not enabled by default on my server.

    I disabled the wp-cron.php in WordPress and setup manual cron job for wp-cron.php once per hour every sixth hours, so the delays should no longer be higher than a six hour frame and the snapshots should run on the specific days they were scheduled to run (not earlier as it happened).

    Regarding Google Drive that really amazes me, since it took hours (with the support on the chat, I even believe you were the one assisting me on the chat) following every single step to be able to get this project / folder authorized and still after that it shows an error. Very frustrating to say the least. I will try to set it up once again and if it fails I would like to set a temporary password on my Google Account to let you guys give it a try. Could we do that if necessary?

    Access was enabled again at WP dashboard and what I meant by

    7) When I deleted a Snapshot from my WP Panel it didn't delete the respective file from my server.

    Is that once I deleted through WP dashboard a snapshot backup created by the plugin, I was expecting that this particular file would be deleted from my cPanel as well, just like it may happen once a new backup overrides the oldest one. But I was surprised to notice that the file I had deleted from the WP dashboard was still at my cPanel, so actually it didn’t help deleting it from the Plugin interface.

    With the measures taking place here I will give a few extra tries to Snapshot plugin, but unfortunately due to the resources consumed while running the backups and the other extra issues associated with it, I am considering to deactivate it from my multisites for now and keep following the results of the plugin development to maybe reinstall it again in a near future.

    Thank you very much for all the assistance so far and please update me as soon as you have a new feedback on these issues.

    Best regards,
    Rogerio Barreto

  • Adam Czajczyk

    Hello Rogerio Barreto,

    Thank you for your response!

    Regarding Google Drive that really amazes me, since it took hours (with the support on the chat, I even believe you were the one assisting me on the chat) following every single step to be able to get this project / folder authorized and still after that it shows an error. Very frustrating to say the least. I will try to set it up once again and if it fails I would like to set a temporary password on my Google Account to let you guys give it a try. Could we do that if necessary?

    Yes, of course. Please keep a support access to the site active so we could access it at anytime and you can post such access credentials securely on "WPMU DEV -> Support" page in a text box below the green "Access active for X days" button (this way that information will not be posted anywhere in public; it will remain within your site and we can only see it after accessing your site via support access - which can be revoked at any time by you, if necessary).

    Just let us know here, please, when you want us to check that and the info is there.

    Is that once I deleted through WP dashboard a snapshot backup created by the plugin, I was expecting that this particular file would be deleted from my cPanel as well, just like it may happen once a new backup overrides the oldest one. But I was surprised to notice that the file I had deleted from the WP dashboard was still at my cPanel, so actually it didn’t help deleting it from the Plugin interface.

    The backup file stored locally should then be deleted. I've just double-checked it on my own setup and can confirm that. This might be related to folder/file permissions or is another symptom of a "bigger issue" that you're experiencing.

    Taking into account what you wrote about "considering to deactivate snapshot", I'd like to ask you if in this case could you perhaps setup a staging site that we could use to troubleshoot it? Such a staging site should be an exact copy of the live site and on the same server (e.g. in sub-folder or under sub-domain) and if we cold access it (including server directly) and work on it, this wouldn't interfere your live setup but any solution applied could later be applied to the live site as well.

    Best regards,
    Adam

  • Rogerio Barreto

    Hello Adam,

    I have just granted access for an extra 5 days and saved Gmail credentials there as well, so you might be able to check the Google Drive destination issue as well.

    I tried to regenerate a Snapshot backup from the Daily backup with no success, and although configured a few days ago it seems like no tries to save the backups daily as scheduled happened lately.

    If necessary, I have also installed and activated the plugin WP Crontrol.

    I will keep Snapshot activated for the next 1 or 2 weeks while the issues I am going through might be investigated and hopefully a solution may be found.

    Please let me know if you need anything else.

    Have an awesome weekend!
    Rogerio Barreto

  • Kasia Swiderska

    Hello Rogerio,

    I will escalate this issue with gDrive destination to our Second Line Support Developers - because right now destination is configured correctly, there is no more error - but snapshots are showing all the time "uploading" and nothing shows in the directory in Google Drive.
    Problem is that there is no errors in Snapshot log and in debug.log there is nothing related to sending to gDrive so developers need to check why it is not sending.
    We will keep you posted about results.

    kind regards,
    Kasia

  • Ari

    Hey Rogerio Barreto

    Hope you are fine!

    Its seems everything is fine except the cron task you set it in the cPanel. The command will be like:
    /usr/bin/curl --user-agent cPanel-Cron https://domain.com.br/wp-cron.php?doing_wp_cron
    And the schedule for 4 times in a day will be like: (0 0,6 * * *).
    I tested it with every one minute: (* * * * *) and it worked for me.

    Now I set the cron job correctly in your cPanel as 4 times in a day.

    Hope it will work fine now, let me know if that not work.

    Regards,
    Ari

  • Rogerio Barreto

    Hi Ari,

    Thanks for the feedback, seems like we are almost there!

    Now the snapshots are being uploaded to Google Drive as expected and as scheduled :slight_smile:

    I will keep monitoring its behavior for the next days. The only thing that seems to not have worked as expected was the exclusion of the server storage backup. The reason I say that is because even after the file has been uploaded to Google Drive it was not deleted from my server and therefore keeps consuming disk usage space.

    I don't know when it is suppose to be deleted after the file is uploaded to Google Drive, but one of them was uploaded to my Google Drive folder 36 hours ago and it still remains saved at my server as well.

    [image pos="0"]

    I believe this is the last issue to solve in order to resolve this thread.

    I have granted support access with the necessary credentials saved in case you guys need to further investigate that.

  • Adam Czajczyk

    Hello Rogerio Barreto

    I don't know when it is suppose to be deleted after the file is uploaded to Google Drive, but one of them was uploaded to my Google Drive folder 36 hours ago and it still remains saved at my server as well.

    In fact, it won't be deleted from your local drive. There's always at least 1 local copy. When you edit the settings of a given snapshot, there's a "Storage Limit" option which currently is set to 4 for your "KAZAS WEEKLY SNAPSHOTS".

    The 4 value doesn't actually mean that there'll be only 4 backup archives stored on your Google Drive - as these are not limited. Instead, it means that for this particular Snapshot up to 4 copies will be stored locally (in additional to Google Drive) and then rotated - so if 5th is made, the oldest one will be removed and still 4 will be kept.

    You can minimize the number of copies stored locally by setting the value in that option to 1 so then only 1 local copy per each configured snapshot will be stored.

    Kind regards,
    Adam