How to optimize WP/VPS?

I'm really confused as to what's causing this, but my VPS is suddenly running at 100% CPU most of the time. The main culprits are both WordPress php files: index.php and wp-cron.php on my main Multisite. These weren't a problem in the past, so I'm confused as to why they suddenly are an issue. I'm working with my host's support, but they're Linux/Hosting experts, not necessarily WordPress experts. So I hoped somebody here might have some ideas.

  • aecnu

    Greetings Tevya Washburn,

    Sorry to see that you are having an unexplainable issue there on your hosting account.

    Though your screen shots are clearly in the post above, they do not give a clue as to why.

    I also admit that on all of my 27 servers, to include 300+ customers and many running several MultiSites, not one is experiencing this phenomenon nor anything close.

    Your hosting company indeed has root access and should be able to pin this down pretty quick.

    However, you can help them by doing standard troubleshooting protocol to help limit the possibilities or perhaps hone in on what plugin is causing the issue if it is indeed a plugin.

    Standard plugin trouble shooting protocol - first to switch to the Twenty Eleven theme just long enough to check for the problem - if the problem still exists next involves deactivating all plugins except the plugin in question and then see if the issue still exists.

    If it does not, then you want to activate plugins one at a time testing in between to see if the issue returns. Even when you find one plugin, it may be in your interest to deactivate the problem plugin and continue testing the rest of the plugins to insure no others are also conflicting.

    You will know the conflict when the issue returns and which plugin(s) you activated that cause the issue.

    Please advise if there are any plugin conflicts and if so what the plugins are that are conflicting.

    Cheers, Joe

  • Tevya

    Okay, we turned off wp-cron and are now setting it up to pull each sub-site's wp-cron through a real cron. That brought down the processes significantly. Where totals used to be around 100% of CPU usage before, now they're around 4%. So that seems to have made a huge difference. However, my overall CPU usage still is still up around 40% - 60% most of the time. If WP (and processes) is no longer the cause, where else could/should I look for what's hogging all the CPU?

  • Connectivity.Engineer

    There are a few things that should be done.

    1. Ask the host for a break down from this command at root:

    $ sar

    Generally that will give us a breakdown of what the server is doing from a I/O level.

    2. Ask them to run a repair on the database. This is not destructive - but should help if something is wrong at that level.

    3. Install the following plugin: WordPress System Health this will give you a general overview of how wordpress is running by itself.

    4. if all else fails - feel free to hit me up off list - we could take a copy of what your running and run it with some strace (system traces) and see if it is an issue or not.

    5. Ask your provider to double check and see if there is an apache/php memory leak - these can happen every so often - and a simple recompile will do wonders.

    6. Is this a managed or self managed vps? Do you have a control panel?
    your control panel may help lead into some answers as well.

    7. How large is your installation? It may help to actually break down and use the multi-database plugin.

    8. Ask your host to review the my.cnf settings. If for example all of innodb is in 1 flat file vs being broken down per table - that could account for some huge latency. There is an easy fix however to that issue (change a setting in my.cnf and then dump the database into a new db or even better use the mutli-database plugin to do it for you )

    Sorry for the many questions - but know what we have seen before - and just thinking off the top of my head possible scenarios that could cause the issue

  • Tevya

    To answer @aecnu's question they responded:

    Showing top on your server with the highest processes and the mysql load as requested:
    Tasks: 60 total, 7 running, 53 sleeping, 0 stopped, 0 zombie
    Cpu(s): 88.3%us, 11.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 2304.000M total, 942.484M used, 1361.516M free, 0.000k buffers
    Swap: 0.000k total, 0.000k used, 0.000k free, 0.000k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    22416 fiddlers 18 0 200m 54m 9348 R 52.7 2.4 0:00.53 /usr/bin/php /home/fiddlers/public_html/wp-login.php
    22412 mytechs 19 0 181m 32m 9140 R 26.9 1.4 0:00.27 /usr/bin/php /home/mytechs/public_html/index.php
    22403 fiddlers 21 0 188m 41m 9456 R 22.9 1.8 0:00.34 /usr/bin/php /home/fiddlers/public_html/index.php
    22409 fiddlers 19 0 200m 53m 9472 R 21.9 2.3 0:00.43 /usr/bin/php /home/fiddlers/public_html/index.php
    22399 fiddlers 21 0 208m 60m 9752 R 20.9 2.6 0:00.52 /usr/bin/php /home/fiddlers/public_html/index.php
    22418 southwri 19 0 165m 31m 8332 R 20.9 1.4 0:00.21 /usr/bin/php /home/southwri/public_html/index.php
    5172 mysql 15 0 495m 311m 7104 S 1.0 13.5 21:06.74 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/fid.fiddlerstud

    @cloudhopping I appreciate it. I need to figure something out, so no need to apologize. Here's the answers I can provide, the rest I've passed on to my hosting provider:

    2. There was a full repair of all databases (and, I think, optimization) done just recently. But I've passed on your suggestion again.

    3. I'll do that.

    4. We'll try this stuff and it it doesn't work, I'd appreciate any help you can provide.

    5. It's sorta both. I have full root access, etc, but they provide extensive support. So I can install what I want, but have only used stuff that they support and let them install it in most cases. My knowledge just doesn't go deep enough.

    7. What number(s) do you want to decide the "size"? There's around 40 sites. Most are just company websites for small businesses and have 8-10 total pages/posts. A few are a little larger, with estores adding a bunch of custom posts, etc. Then a few are larger with blogs that and are posted on regularly or have a large product selection in their estore. But none are anything near what I'd consider massive or huge by most standards.

    I'm also running a few other Multisites and single-site installs on this server. But none are currently over 5 sites, etc.

  • Connectivity.Engineer

    I just posted this in another thread - but one BIG question for your host -

    Is the database setup to use Innodb OR myisam?

    MyISAM locks tables on each write - and that can slow stuff down quite heavily ...

    if you can dump the /etc/my.cnf file here - (scrub for passwords first please !!! )

    that will help us help you

    88% cpu is a bit high btw...

    Is this a cpanel server?
    If so - ask them to install Apachebooster http://applications.cpanel.net/apachebooster-4/

    ApacheBooster is supported on other platforms as well - so your host still should be able to support you utilizing it.

    It is simply amazing what Varnish + Nginx in front of your apache install will do.

    if you can do a dump of SAR as well for a 24 hour period that will help us help you as well.

  • Tevya

    Thanks for this. I've passed these on to my host as well. I'd previously heard that Nginx was great, but when I asked Hostgator, they said they didn't support it. So I could install it, but they wouldn't and wouldn't support it for any related problems. I've included you suggestion of Apachebooster anyway, sometimes I receive different answers from different techs.

    They'd previously provided an SAR dump and I forgot to post it back here. Here it is:

    root@fid [/home/fiddlers/public_html]# sar
    Linux 2.6.18-028stab093.2 (fid.fiddlerstudios.com) 03/27/2013

    12:00:01 AM CPU %user %nice %system %iowait %steal %idle
    12:10:01 AM all 86.50 0.18 13.32 0.00 0.00 0.00
    12:20:01 AM all 86.39 0.35 13.25 0.00 0.00 0.00
    12:30:05 AM all 85.96 0.19 13.85 0.00 0.00 0.00
    12:40:01 AM all 85.68 0.31 14.01 0.00 0.00 0.00
    12:50:04 AM all 85.67 0.19 14.13 0.00 0.00 0.00
    01:00:01 AM all 86.42 0.02 13.56 0.00 0.00 0.00
    01:10:05 AM all 86.28 0.29 13.43 0.00 0.00 0.00
    01:20:07 AM all 86.17 0.22 13.60 0.00 0.00 0.00
    01:30:08 AM all 86.07 0.19 13.74 0.00 0.00 0.00
    01:40:02 AM all 86.34 0.05 13.61 0.00 0.00 0.00
    01:50:01 AM all 86.58 0.03 13.39 0.00 0.00 0.00
    02:00:04 AM all 86.89 0.03 13.06 0.02 0.00 0.00
    02:10:01 AM all 86.90 0.02 13.07 0.00 0.00 0.00
    02:20:01 AM all 85.84 0.17 13.75 0.11 0.00 0.13
    02:30:02 AM all 86.24 0.32 13.42 0.02 0.00 0.00
    02:40:03 AM all 86.49 0.21 13.30 0.00 0.00 0.00
    02:50:05 AM all 86.39 0.21 13.39 0.00 0.00 0.00
    03:00:07 AM all 86.03 0.31 13.65 0.00 0.00 0.00
    03:10:02 AM all 86.39 0.08 13.52 0.00 0.00 0.00
    03:20:01 AM all 86.23 0.02 13.75 0.00 0.00 0.01
    03:30:03 AM all 86.39 0.02 13.59 0.00 0.00 0.00
    03:40:02 AM all 86.19 0.04 13.77 0.00 0.00 0.00
    03:50:02 AM all 85.64 0.01 13.84 0.00 0.00 0.51
    04:00:02 AM all 86.31 0.02 13.65 0.00 0.00 0.03
    04:10:13 AM all 85.97 0.01 14.02 0.00 0.00 0.00
    04:20:03 AM all 85.96 0.07 13.97 0.00 0.00 0.01
    04:30:03 AM all 86.20 0.20 13.61 0.00 0.00 0.00
    04:40:01 AM all 86.45 0.15 13.40 0.00 0.00 0.00
    04:50:01 AM all 86.39 0.23 13.38 0.00 0.00 0.00
    05:00:01 AM all 86.08 0.22 13.70 0.00 0.00 0.00
    05:10:01 AM all 85.81 0.28 13.91 0.00 0.00 0.00
    05:20:02 AM all 86.00 0.16 13.84 0.00 0.00 0.00
    05:30:08 AM all 86.00 0.01 13.98 0.00 0.00 0.00
    05:40:12 AM all 85.93 0.02 14.04 0.00 0.00 0.01
    05:50:05 AM all 85.94 0.02 14.02 0.00 0.00 0.03
    06:00:01 AM all 85.99 0.01 13.98 0.00 0.00 0.02
    06:10:01 AM all 85.53 0.02 14.43 0.00 0.00 0.02
    06:20:03 AM all 85.82 0.60 13.52 0.00 0.00 0.06
    06:30:01 AM all 86.51 0.53 12.95 0.00 0.00 0.01
    06:40:01 AM all 86.41 0.02 13.56 0.00 0.00 0.01
    06:50:01 AM all 86.46 0.02 13.50 0.00 0.00 0.03
    07:00:02 AM all 86.45 0.01 13.52 0.00 0.00 0.01
    07:10:02 AM all 86.40 0.01 13.56 0.00 0.00 0.02
    07:20:02 AM all 86.55 0.07 13.37 0.00 0.00 0.01
    07:30:01 AM all 86.46 0.10 13.44 0.00 0.00 0.00
    07:40:01 AM all 86.67 0.14 13.19 0.00 0.00 0.00
    07:50:03 AM all 86.64 0.00 13.35 0.00 0.00 0.00
    08:00:03 AM all 76.29 0.02 12.20 0.48 0.00 11.01
    08:10:01 AM all 86.43 0.17 13.26 0.00 0.00 0.14
    08:20:06 AM all 85.81 1.01 13.12 0.00 0.00 0.06
    08:30:01 AM all 86.40 0.01 13.49 0.00 0.00 0.09
    08:40:01 AM all 86.52 0.02 13.35 0.00 0.00 0.12

    08:40:01 AM CPU %user %nice %system %iowait %steal %idle
    08:50:01 AM all 86.20 0.03 13.67 0.00 0.00 0.10
    09:00:02 AM all 86.17 0.02 13.75 0.00 0.00 0.06
    09:10:01 AM all 86.36 0.02 13.57 0.00 0.00 0.05
    09:20:02 AM all 85.72 0.01 14.17 0.00 0.00 0.09
    09:30:01 AM all 85.40 0.01 14.46 0.00 0.00 0.13
    09:40:01 AM all 85.67 0.02 14.16 0.00 0.00 0.14
    09:50:01 AM all 85.54 0.03 14.35 0.00 0.00 0.08
    10:00:02 AM all 85.49 0.02 14.37 0.01 0.00 0.11
    10:10:03 AM all 85.40 0.01 14.55 0.00 0.00 0.04
    10:20:01 AM all 85.48 0.66 13.81 0.00 0.00 0.05
    10:30:01 AM all 85.66 0.64 13.69 0.00 0.00 0.01
    10:40:01 AM all 86.42 0.02 13.54 0.00 0.00 0.02
    10:50:01 AM all 86.11 0.01 13.84 0.00 0.00 0.04
    11:00:01 AM all 86.11 0.02 13.84 0.00 0.00 0.04
    11:10:08 AM all 86.23 0.02 13.71 0.00 0.00 0.03
    11:20:02 AM all 85.86 0.02 14.08 0.00 0.00 0.05
    11:30:01 AM all 85.73 0.03 14.23 0.00 0.00 0.02
    11:40:03 AM all 85.19 0.01 14.78 0.00 0.00 0.02
    11:50:07 AM all 85.85 0.01 14.13 0.00 0.00 0.01
    12:00:02 PM all 85.96 0.02 13.98 0.01 0.00 0.04
    12:10:08 PM all 84.97 0.02 15.00 0.00 0.00 0.00
    12:20:02 PM all 85.90 0.32 13.77 0.00 0.00 0.01
    12:30:10 PM all 86.53 0.53 12.91 0.00 0.00 0.03
    12:40:02 PM all 85.99 0.49 13.52 0.00 0.00 0.00
    12:50:02 PM all 86.10 0.02 13.87 0.00 0.00 0.01
    01:00:11 PM all 86.03 0.02 13.94 0.00 0.00 0.01
    01:10:02 PM all 86.14 0.02 13.84 0.00 0.00 0.01
    01:20:01 PM all 86.24 0.02 13.69 0.00 0.00 0.05
    01:30:01 PM all 85.08 0.01 14.07 0.01 0.00 0.82
    01:40:06 PM all 84.85 0.01 14.08 0.01 0.00 1.04
    01:50:01 PM all 85.21 0.03 14.07 0.00 0.00 0.69
    02:00:01 PM all 78.95 0.02 13.82 0.03 0.00 7.19
    02:10:01 PM all 77.10 1.11 13.21 0.07 0.00 8.51
    02:20:01 PM all 77.73 0.02 12.78 0.05 0.00 9.42
    02:30:01 PM all 79.37 0.01 13.03 0.08 0.00 7.50
    02:40:01 PM all 78.72 0.04 12.76 0.05 0.00 8.42
    02:50:01 PM all 79.51 0.02 13.22 0.03 0.00 7.22
    Average: all 85.45 0.13 13.68 0.01 0.00 0.72

    =====================================

    To be clear on a few things:

    1. We upgraded the VPS another level, so it now has 2-cores allocated, and a lot more ram. The RAM seems steady at about half the total, so that's well under control.
    2. We turned off wp-cron on the largest multisite and scheduled actual cron jobs to fire each site's wp-cron on a schedule, and now wp-cron rarely shows up as utilizing any processor.
    3. In-spite of the above, CPU is still at 95 - 100% whenever I check on it. However, I did install WordPress System Health and it shows the CPU as using around 10% most of the time with spikes up to 20%. I assume that's just the part which that particular WP install is using, and not the same as what PPP is showing me....?

  • Tevya

    @cloudhopping here's what they responded (my comments in parenthesis):

    When I accessed your server today, the user fiddlers was again causing significant load problems. Here is an example of the processes that were running causing the load:

    fiddlers 16149 23.5 2.5 209740 59748 ? R 13:44 0:00 \_ /usr/bin/php /home/fiddlers/public_html/index.php
    fiddlers 16165 30.0 1.7 192012 41704 ? R 13:44 0:00 \_ /usr/bin/php /home/fiddlers/public_html/index.php

    With regards to ApacheBooster, I am willing to install this for you, but it is not supported. If you do want me to install this, I cannot guarantee that it will work, and we also will not be able to support any aspect of this software after the installation. We cannot provide post-installation configuration, troubleshooting, or debugging and also important to consider is that I cannot guarantee we can remove the installation if you change your mind later. (So I'm kinda hesitant, since I rely on them heavily, since my knowledge of that end is limited)

    The database engine for fiddlers WordPress installation is MyISAM. I wouldn't expect any significant performance increases by converting the database to innodb. Currently, even when your server is under very high load, mysql is processing well:

    14:06:36 up 6 days, 14:41, 1 user, load average: 13.10, 15.73, 16.36

    mysql 5172 1.1 13.4 509260 317536 ? Sl Mar26 42:43 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/m

    root@fid [/home/fiddlers/public_html]# mysqladmin pr
    +--------+----------------+-----------+----------------+---------+------+-------+------------------+
    | Id | User | Host | db | Command | Time | State | Info |
    +--------+----------------+-----------+----------------+---------+------+-------+------------------+
    | 735333 | fiddlers_wrdp1 | localhost | fiddlers_wrdp1 | Sleep | 1 | | |
    | 735334 | fiddlers_wrdp1 | localhost | fiddlers_wrdp1 | Sleep | 1 | | |
    | 735336 | fiddlers_wrdp1 | localhost | fiddlers_wrdp1 | Sleep | 0 | | |
    | 735337 | fiddlers_wrdp1 | localhost | fiddlers_wrdp1 | Sleep | 0 | | |
    | 735338 | fiddlers_wrdp1 | localhost | fiddlers_wrdp1 | Sleep | 0 | | |
    | 735339 | fiddlers_wrdp1 | localhost | fiddlers_wrdp1 | Sleep | 0 | | |
    | 735340 | southwri_prod | localhost | southwri_prod | Sleep | 0 | | |
    | 735341 | fiddlers_wrdp1 | localhost | fiddlers_wrdp1 | Sleep | 0 | | |
    | 735342 | root | localhost | | Query | 0 | | show processlist |
    +--------+----------------+-----------+----------------+---------+------+-------+------------------+

    It is not locked tables causing problems, it is the amount of cpu resources required for each page load of the site that is a problem. I have included below the contents of your my.cnf file.

    [mysqld]
    innodb_file_per_table=1 # Ensure that each innodb table is it's own binary data block just in case there's corruption.
    query_cache_size=64M
    thread_cache_size=64 # can be increased on servers with large numbers of active users
    key_buffer_size=32M
    max_allowed_packet=16M # don't change unless required for large blobs
    table_cache=2048 # max 2048, can be increased if more Opened tables - SHOW STATUS LIKE 'Opened_tables';
    table_definition_cache=8192 # increase by the same factor as table_cache
    wait_timeout=20 # can be increased if using persistent connections
    max_user_connections=25
    innodb_flush_method=O_DIRECT
    open_files_limit=31004

    #delayed_insert_timeout=20 # Turn on if max_connections being reached due to delayed inserts
    #delayed_queue_size=300 # Turn on if max_connections being reached due to delayed inserts

    myisam_sort_buffer_size=2M # can be increased per sessions if needed for alter tables (indexes, repair)

    #query_cache_limit=2M # leave at default unless there is a good reason
    #join_buffer=2M # leave at default unless there is a good reason
    #sort_buffer_size=2M # leave at default unless there is a good reason
    #read_rnd_buffer_size=256K # leave at default unless there is a good reason
    #read_buffer_size=2M # leave at default unless there is a good reason
    collation_server=utf8_unicode_ci
    character_set_server=utf8

    #general_log=1
    slow_query_log=1
    log-output=TABLE # select * from mysql.general_log order by event_time desc limit 10;
    long_query_time=20 # select * from mysql.slow_log order by start_time desc limit 10;

    tmp_table_size = 128M
    max_heap_table_size = 128M
    innodb_buffer_pool_size=8M # check mysql -e "SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool%';" - free vs total
    innodb_thread_concurrency=4 # Number of physical + virtual CPU's, be careful of adding more
    max_connections=100 # Should be between 100-150, increase *slowly* because it causes MySQL to consume more memory!

    wait_timeout=45
    connect_timeout=20

  • Connectivity.Engineer

    ok - so the good news is it is not database related.
    Another piece of good news is that it is not (at least on first review) a disk I/O issue.

    Is this a cpanel server?
    if so - there are a few things that can be done - you could "trace" a whm thread from within the whm cpu resources overview

    There are a number of reasons high cpu from "apache" could happen.
    You might want to ask them to run this command as well

    netstat -lpn|grep :80 |awk ‘{print $5}’|sort

    If a particular IP has more than say 25 hits at the same time - chances are something is out of the ordinary coming from that IP

    You might also want to use this plugin to check what might be causing the load

    http://wordpress.org/extend/plugins/p3-profiler/

    In short - index.php is the entry file for Wordpress but it can be misleading because most everything runs as index.php in one way or another...

    I will give you a call as well - to try to help sort things out.

  • aecnu

    Greetings Tevya Washburn and cloudhopping,

    I totally agree with cloudhopping's observation that it is not disk I/O nor database related which both show to be none existent issues as shown by the screen shots above i.e. zero wait state and zero database load.

    However, this is troubling in itself because one would expect if these are indeed WordPress installations that there would be both database activity and I/O as it is reading the pages as Host Gator claims.

    Keep us posted as to what you find out, it is definitely an intriguing symptom.

    Look forward to hearing from you folks in any event.

    Cheers, Joe

  • Tevya

    @aecnu thanks for following up. @cloudhopping and I worked on this outside these forums and he was able to further diagnose the problem. He's been incredibly helpful. I've actually moved to his hosting now, and he's been taking great care of me. I'm not sure whether or not he wants me talking about his company here, but I can't say enough good about them.

    They've discovered a bunch of things that HostGator could not (or wouldn't take the time to), including that my VPS had been compromised via a recent exploit in cPanel. We're working together to get everything sealed back up, but that seems to be one of the major causes of a lot of the trouble.

    Thanks to all his help, we should have it all extra-secured in the very near future. But my sites are running incredibly well and I'm able to keep working ever since they finished up the migration and some optimization. So I'm marking this as resolved. Thank you.

  • aecnu

    Greetings Tevya Washburn,

    Thank you for letting us know and happy to see that you got this resolved in any event.

    I do not see anything wrong in talking about him or his company and I love and thrive on competition :wink: especially those that expose things in truth for what they are - in this case Host Gator incompetence - exploits and they do not even know it? Even though they were advised there was an issue! Unbelievable.

    Cheers, Joe

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.