How to optimize server for a multisite?

Hello folks,

I have been having some over usage issues on my server where my mulitsite resides. This is on a shared server and I will eventually move to a dedicated server when I have enough paying members.

I have able to discover some of the problems during the past few months that were straight-forward, for example, a couple versions back, the Domain Mapping plugin was causing a script to be continuously running, which resulted in over usage and shut down my server.

But, lately, I have been getting these spikes of over usage which is always accompanied by too many processes running at once.

Sorry for the long email, but I want to provide enough background information. I also have a screenshot of my server usage, but I don't see where to upload the image.

Here are the last recommendations from my hosting company (Hostgator), which I haven't implemented yet. I would like to ask your opinion if you think that implementing the part about adding a Cron Job would be a good idea. Thanks, Joe.

------------------------------------------------------------------------------

Hello Joe,

I have included your reply in my reply to be able to make sure that all of my answers are associated with the context.

Earlier today (2.26.15) at 11:00, my CPU usage graph showed a spike of
about 3595 seconds, which caused my server to be put under the
"restriction" again. I happen to be in my Cpanel when this was happening. What I noticed was that 26 processes were running. I believe this caused the over usage. All the processes were referencing
.../public_html/index.php files. I think this is my primary website at
wordsandwriters.com. Was this spike a hacker trying to brute-force into may
site?

The way that WordPress is designed all requests are routed through the index.php file. This is normal behavior. That said, I did see Google crawling your site at that time.

All the other reps could tell me is that it represented "traffic." I am
trying to determine what it was, so I can make any necessary changes to my website. They said you could look at a more detailed level to help me
troubleshoot this.

The other thing I need to understand is that when I look at my awstats
summary of the top 25 hosts, I see one that this IP address, 192.185.245.129, has over 135,000 visits this month on wordsandwriters.com. There is also this same IP address shown under *.wordsandwriters.com that has over 16,000
visits.

I did a lookup of this IP address and it appears it is from Hostgator. Why
is the number of visits so high? I know I am not getting actual traffic
from real people of this magnitude. What does this large number of visits represent. Can you look at the logs to give me some clue on whether I can or need to do something about this.

The reason for the large number of requests from this IP address is that the server is making requests to itself for the wp-cron system. You can disable the visit based cron, and instead set up a regular cron which you can have much better control over by doing the following:

Add a line to your wp-config.php that reads

define('DISABLE_WP_CRON', true);

This will prevent visits to index.php from launching the wp-cron process. You are then responsible for launching these processes yourself. But that is not a problem because you can simply create a cron job using cPanel to run no more often than once every 15 minutes to run the following:

wget -q -O /dev/null http://site-name-here.com/wp-cron.php

I am wondering if this large number is adversely affecting my CPU usage, processes running, Inodes anything I need to be concerned about.

In summary:

1. What caused the spike at 11am today?
It appears that quite a bit of this usage was caused by Google Crawling your site. There was also some normal traffic at this time as well.

You may wish to request that Google adjust how they crawl your site. We have directions to do this at:

http://support.hostgator.com/articles/specialized-help/telling-google-how-often-to-crawl-your-website

2. Anything I need to do about the 135, 000 visits from that IP address?

You can disable the visit based cron and set up real cron to handle your WordPress schedueld work
--------------------------------------------------------------------------------------

  • Ash

    Hello @joejacobson

    I hope you are well today.

    First of all, running a multisite with 40+ subsites in a shared hosting is not a good idea. You can buy any small VPS package from Digital Ocean or Ramnode or VPS Dime, they are cheap but good enough. You just need to configure the server in the most optimized way.

    In a shared hosting, you will have lots of limitation. Also, WordPress needs a lot of memory. In your case, I think the suggest way should be helpful for you. Let me explain:

    Adding this line in your wp-config.php:

    define('DISABLE_WP_CRON', true);

    This line will disable wordpress' default cron job. Cron job is an automatic process, that runs after certain interval. So, they suggested you to disable that wordpress cron and instead set a cron job from cPanel in every 15 mins. It will run the same cron file of your WordPress system but in larger interval, so it will minimize some loads on your server.

    Hope it helps :slight_smile: Please feel free to ask more question if you have any.

    Cheers
    Ash

  • joejacobson

    Hi Ash,

    Thanks for your feedback. Yes, I know that I need to upgrade. I checked with my current host and they said that a VPS wouldn't necessarily help me with the performance issues I'm having with too much CPU usage. They suggested a dedicated server. Just a little out of my budget at this moment, but I know I need it soon.

    Regarding setting up the cron job, here's my question:

    If I set it up to run every 15 minutes, will that impact visitors to my site from a browser or my subsite administrators? Let me take one at a time:

    1. General visitors: Will they still be able to visit each page, sign up for a site or click on links and fill out forms even with the cron set? In other words, is there any impact to them?
    2. Will the cron job impact my subsite administrators in their ability to get into their dashboard and do functions like create a new page or post? Can they do this even without the cron being initiated at that moment? Or are these two aspect not related? I'm not clear on how running a cron job will affect normal operations for my main site and subsites.

    Thanks, Joe.

  • Ash

    Hello @Joe

    I hope you are well today.

    In both cases, setting a cron job should not be an issue. There is no relation among creating posts, pages or logging and cron jobs. Setting a cron job from cpanel will reduce the automatic process of wordpress itself. For example, if your wordpress did X things in every 5 mins, now it will do that in every 15 mins. Also, as the control is over cpanel so it will decide based on server load when to run. Note that, it doesn't necessarily mean that you won't see over usage anymore, but it will definitely reduce the load than your current state.

    You also need to rely on some good plugins. Some plugins really need much memory, like buddypress. Even in domain mapping plugin, if you enable SSO it will take more memory too.

    I recommend to give it a try and let's see what happens.

    Cheers
    Ash

  • joejacobson

    Hi Ash,

    I added the cron job, and it seems like it is having a positive effect since it started (4 hours so far). I will keep on eye on it. Today, I received another response from my host support ticket. They recommended some more adjustments to my server that looks even more promising. Here is their reply:
    --------------------------------------------------------------------------------
    "Thank you for contacting HostGator. The cron won't effect people being able to post content or login. However, it looks more like this issue is due to the large number of admin-ajax processes due from your WordPress heartbeat API. I would suggest reviewing this article and disabling the heartbeat API as you have a large amount of people logging into the dashboards of their site.

    http://support.hostgator.com/articles/specialized-help/technical/wordpress/wordpress-heartbeat
    ------------------------------------------------------------------------------------------
    I looked at the support article and it is quite complete and somewhat complicated (for me). The article has several routes you can take depending on how much you want to minimize or turn off the WordPress Heartbeat API.

    I wonder if you could scan over the article and recommend which of their options to take. Thanks, Joe.

  • Ash

    Hello @Joe

    That's definitely another good suggestion. In short, I recommend to increase the heartbeat interval first:

    add_filter( 'heartbeat_send', 'my_heartbeat_settings' );
        function my_heartbeat_settings( $response ) {
                if ( $_POST['interval'] != 120 ) {
                        $response['heartbeat_interval'] = 120; // 2 minutes
                }
                return $response;
        }

    If you still have problem with your host, then you try disabling heartbeat:

    add_action( 'init', 'stop_heartbeat', 1 );
    
        function stop_heartbeat() {
                wp_deregister_script('heartbeat');
        }

    You can add those codes in your functions.php in the theme, if you think your theme won’t be changed. Otherwise mu-plugins is the best solution. To use mu-plugins, go to /wp-content/ and find the folder with name 'mu-plugins'. If there is no folder in that name, then create a folder, name it 'mu-plugins', create a file inside that, give any name you like and paste the code in there. You don't need to activate that plugin. Mu-plugins means must use plugins, so it will be activated automatically always. If you use mu-plugins then add a php start tag at the beginning of the code.

    Hope it helps :slight_smile: Please feel free to ask more question if you have any.

    Cheers
    Ash

  • joejacobson

    Thanks again, Ash. I'm afraid I need a little more specific guidance. In your instructions, you said to "create a file and paste the code in there." What kind of a file am I supposed to create? A php file? Is there a basic template for a php file I can use? Also, can you provide what the php start tag would look like? And can I do all this in my host's file manager?

    Sorry for the fundamental questions. You can tell I am not a programmer.

    Thanks, Joe.

  • Ash

    Hello @joejacobson

    I am sorry I should be more clear.

    Yes, you have to create a normal php file. A php start tag looks like <?php

    You can give any name to that file, but the extension .php. For example, create a blank text file and save as stop_heartbeat.php and then add the following code:

    <?php
    add_filter( 'heartbeat_send', 'my_heartbeat_settings' );
        function my_heartbeat_settings( $response ) {
                if ( $_POST['interval'] != 120 ) {
                        $response['heartbeat_interval'] = 120; // 2 minutes
                }
                return $response;
        }

    or

    <?php
    add_action( 'init', 'stop_heartbeat', 1 );
    
        function stop_heartbeat() {
                wp_deregister_script('heartbeat');
        }

    Then go to /wp-content/ folder. If there is no mu-plugins folder already, just create one and upload the stop_heartbeat.php file inside of mu-plugins folder.

    Hope it helps :slight_smile: Please feel free to ask more question if you have any.

    Cheers
    Ash

  • Ash

    Hello @joejacobson

    I already had a mu-plugins directory. It had a folder named "new folder" already and in that folder was this file: "no-white-screen.php" I don't remember adding that. Do I need this file?

    I am not sure about no-white-screen.php file. Maybe any plugin is creating this file then? Deleting that file without being confirmed is not a good idea I think. So, let it be for now.

    One more thing, the above file should be at root of mu-plugins folder. Do not, put in any folder inside mu-plugins folder.

    Correct: /wp-content/mu-plugins/yourfile.php
    Wrong: /wp-content/mu-plugins/any-folder/yourfile.php

    And sure, let me know how it goes.

    Cheers
    Ash

  • joejacobson

    Hi Ash,

    Give giving you a report after one week of server optimization. Everything seems to be stable now with few spikes of CPU usage and even those spikes are not severe. I've attached the before and after images of my CPU usage. The main changes that seemed to help were the cron-job (at 15 min) and the change in the Heartbeat API interval (120 seconds).

    Thanks again for you help. This has been a life-saver until I can switch to a dedicated server. And I did all these changes on my shared server.

    Joe.

  • joejacobson

    Hi Ash,

    Back again still working on server optimization. As you can see from the last reply, the second image of CPU usage went way down with no spikes. It continued like this for about a week and then started climbing again, though no huge spikes like before.

    I tried changing the cron job to 30 minute intervals and also tried stopping the Heartbeat API completely. Maybe those weren't the best options to try. I noticed that my snapshots are now delayed when they run by 30 minutes, but they do run. But I was wondering if snapshots is sending a request to the server and using resources for 30 minutes until the cron job kicks in. I don't know how this works. I could do the snapshots manually, but I have 40 subsites now and plan to have many more in the future.

    A couple more related questions:
    1. You mentioned SSO settings in the Snapshot plugin. Where is that setting?
    2. In the compresssion library section, it offers ZipArchive or PclZip. Which would be a better choice with regard to CPU usage?
    3. I also have a warning message: "Warning: Your Snapshot folder is publicly accessible. Please make sure you DO NOT use the default "snapshot" location. Although we attempt to disallow directory listing and are obscuring your backup names, having this folder public opens it up for exploitation. You may need to consult your hosting provider to move this folder outside of your website folder. Example path:disappointed:home1/josen/snapshots" Is this important to change?
    4. Regarding the segment size for Snapshot: it is now set to 1000. Would 2000 be better with regard to CPU usage?
    5. I noticed that my [/feed] is listed in my traffic with very high hits on my site. Is there any way to minimize this impact?
    6. My host company also recommended setting up a robots.txt file using this code:

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/
    Disallow: /trackback/
    Disallow: /index.php
    Disallow: /xmlrpc.com
    Disallow: /wp-login.php
    Disallow: /wp-content/plugins/
    Disallow: /comments/feed/
    User-agent: Yandex
    Disallow: /
    User-agent: Baiduspider
    Disallow: /

    Thanks, Joe.

  • Ash

    Hello @joejacobson

    I hope you are well today.

    1. You mentioned SSO settings in the Snapshot plugin. Where is that setting?

    I mentioned SSO settings in the domain mapping plugin, not the snapshot plugin. If you have domain mapping activated, you can go to Network Admin > Settings > Domain Mapping and disable cross domain auto login.

    2. In the compresssion library section, it offers ZipArchive or PclZip. Which would be a better choice with regard to CPU usage?

    ZipArchive would be helpful in this case.

    3. I also have a warning message: "Warning: Your Snapshot folder is publicly accessible. Please make sure you DO NOT use the default "snapshot" location. Although we attempt to disallow directory listing and are obscuring your backup names, having this folder public opens it up for exploitation. You may need to consult your hosting provider to move this folder outside of your website folder. Example path:disappointed:home1/josen/snapshots" Is this important to change?

    If you make this changes, then your snapshots won't be accessible to the outer world. As your snapshot folder is public, so if anyone knows the path they can download the snapshots backup archive. If you are shared hosting, I don't think you can make it private though.

    4. Regarding the segment size for Snapshot: it is now set to 1000. Would 2000 be better with regard to CPU usage?

    Well, that depends on your host. If you increase the value here, it will need more execution time and yes, it will take more memory too.

    5. I noticed that my [/feed] is listed in my traffic with very high hits on my site. Is there any way to minimize this impact?

    I didn't understand this question, would you please explain? Maybe rephrase?

    6. My host company also recommended setting up a robots.txt file using this code:

    That's a good suggestion, you can try that.

    Hope it helps :slight_smile: Please feel free to ask more question if you have any.

    Cheers
    Ash

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.