Troubleshooting: High Load Issue

Where is a good place to start to troubleshoot high load spikes? We have a specific blog that seems to trigger loads of 2 to 6% or more when the page is re-loaded a couple of times... this still occurs will all normal plugins (not mu-plugin folder) disabled and occurs with different themes as well.

This blog isn't huge, 70 or so posts, but was recently imported into MU from a single intall.. is there a chance that the import may have been corrupted or something similar? Grapsing for straws here....

WHM is showing the following processes seem to be involved:

Top Process %CPU 54.0 /usr/bin/php /home/tracedef/public_html/index.php
Top Process %CPU 53.0 /usr/bin/php /home/tracedef/public_html/index.php
Top Process %CPU 45.0 /usr/bin/php /home/tracedef/public_html/wp-cron.php

    Tracy

    Thanks, I've got that, but still need to resolve root issue....

    I'm noticing that even navigating around my dashboard will rock my CPU load to 5 or 6 percent... which isn't normally humanly possible, not even when gzip backups are running... the highest natural load I've EVER seen is around 1 percent and gzips might hit 1.5 to 2.... I've only seen anything over 2.5 if something is wrong....

    Is it possible its time to set up multi-dbs based on these numbers?:

    Total: 331 Tables 24,908 Records

    Tracy

    Thanks! Turns out slow queries log was not enabled, but is now.

    Also noticing that accessing specific blog dashboards brings server to a crawl, even by simply accessing first page of the specific blog's dashboard via blogs > backend from main blog dashboard..... other blogs don't have the issue... for a second it seemed as though changing privacy options from only allowing admins to see the blog to public helped, but could have been coincidence.

    I also noticed the following in my error log from earlier today, first time this series of errors appears that I've ever seen and it pretty much occurs over and over. Not sure if time syncs up exactly with when the issue first started...but cpu usage seems to be consistently related to the following (last two hours average) :

    Top Process %CPU 39.0 /usr/bin/php /home/tracedef/public_html/wp-cron.php
    Top Process %CPU 36.0 /usr/bin/php /home/tracedef/public_html/wp-login.php

    Error Log (sorry for posting all of this, hoping something might stick out or be helpful at that specific time):

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = 'active-member-theme' AND site_id = 1 made by require, require_once, require_once, get_template_directory, get_template, apply_filters, call_user_func_array, bp_core_force_buddypress_theme, get_site_option
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = 'active-member-theme' AND site_id = 1 made by require, require_once, require_once, get_stylesheet_directory, get_stylesheet, apply_filters, call_user_func_array, bp_core_force_buddypress_stylesheet, get_site_option
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = 'automessage_installed' AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, get_site_option
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query show tables like 'wp_am_actions' made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SHOW TABLES; made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query CREATE TABLE wp_am_actions (
    id bigint(20) NOT NULL auto_increment,
    level varchar(10) default NULL,
    action varchar(150) default NULL,
    title varchar(250) default NULL,
    description text,
    PRIMARY KEY (id),
    KEY level (level),
    KEY action (action)
    ) made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SHOW TABLES; made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query CREATE TABLE wp_am_schedule (
    id bigint(20) NOT NULL auto_increment,
    system_id bigint(20) NOT NULL default '0',
    site_id bigint(20) NOT NULL default '0',
    blog_id bigint(20) NOT NULL default '0',
    action_id bigint(20) default NULL,
    subject varchar(250) default NULL,
    message text,
    period int(11) default '0',
    timeperiod varchar(50) default 'day',
    pause tinyint(4) default '0',
    PRIMARY KEY (id),
    KEY system_id (system_id),
    KEY site_id (site_id),
    KEY blog_id (blog_id),
    KEY action_id (action_id)
    ) made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SHOW TABLES; made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query CREATE TABLE wp_am_queue (
    id bigint(20) NOT NULL auto_increment,
    schedule_id bigint(20) default NULL,
    runon bigint(20) default NULL,
    sendtoemail varchar(150) default NULL,
    user_id bigint(20) NOT NULL default '0',
    blog_id bigint(20) default '0',
    site_id bigint(20) default '0',
    PRIMARY KEY (id),
    KEY action_id (schedule_id),
    KEY runon (runon),
    KEY user_id (user_id),
    KEY blog_id (blog_id),
    KEY site_id (site_id)
    ) made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query INSERT INTO wp_am_actions (level,action,title) VALUES ('site','wpmu_new_blog','Create new blog') made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query INSERT INTO wp_am_actions (level,action,title) VALUES ('blog','wpmu_new_user','Create new user') made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = 'automessage_installed' AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, update_site_option
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = 'automessage_installed' AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, update_site_option, add_site_option
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query INSERT INTO wp_sitemeta (site_id,meta_key,meta_value) VALUES ('1','automessage_installed','yes') made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, update_site_option, add_site_option
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT ID, post_name, post_parent FROM wp_160_posts WHERE post_type = 'page' made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, automessageadmin->flush_rewrite, WP_Rewrite->flush_rules, WP_Rewrite->wp_rewrite_rules, WP_Rewrite->rewrite_rules, WP_Rewrite->page_rewrite_rules, WP_Rewrite->page_uri_index
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT ID, post_name, post_parent FROM wp_160_posts WHERE post_type = 'page' made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->flush_rewrite, WP_Rewrite->flush_rules, WP_Rewrite->wp_rewrite_rules, WP_Rewrite->rewrite_rules, WP_Rewrite->page_rewrite_rules, WP_Rewrite->page_uri_index
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = 'map_supporteronly' AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, domain_map->setup_plugin, get_site_option
    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT option_value FROM wp_160_options WHERE option_name = 'rich_editing' LIMIT 1 made by require, require_once, require_once, do_action, call_user_func_array, cforms_addbuttons, get_user_option, get_option
    [20-Oct-2009 19:30:22] Wor

    Tracy

    Thanks Barry, I will definitely check it out. I'm wondering if the mysql has gone away is more an after effect of server resources not being available due to other runaway services or processes....

    I think the privacy-settings plugin is a suspect at some role in what is going on. One thing I've noticed that is consistent: If I log into the dashboard of a blog that has the privacy settings set to "I would like only administrators of this blog, and network to see my blog." then the server starts to lock up. If I change the privacy settings to "I would like blog to be visible to everyone..." then the CPU load immediately goes back to normal and the problem goes away. Then I am unable to change back to the previous setting of "I would like only administrators of this blog, and network to see my blog." and the CPU issue does not occur.... at least for a while....

    I looked at privacy settings as a cause because only two blogs have this issue and they are both blogs we recently imported that will replace single installs. Since we haven't mapped them yet and are doing work on them, they are set to the strictest privacy setting until we map them......

    Andrew

    Hmm, I wonder if it's the redirect causing the problem. Try changing it to the troublesome privacy option and comment out line 68 in the plugin. That line handles the redirect for that privacy option and is really the only bit of code that could cause any problems.

    What's weird is that a ton of people use that plugin and we've never had a report of any problems. Your site is actually fairly small too and it's in use on much larger sites. We're not experiencing any problems on Edublogs/Campus either.

    Thanks,
    Andrew

    Tracy

    Andrew, I have commented that line out and will follow up if I see the issue occur again.... it's a little hard to test as the issue is a bit inconsistent as once we change privacy settings from "I would like only administrators of this blog, and network to see my blog." to "Anyone can see blog" and back, the issue goes away for a while... There seems to be some window after this change where the issue doesn't occur ...

    Thank you so much for the feedback and help guys!

    Tracy

    Quick update: I just logged into one of the two blogs affected and it very apparent when the load problem exists because logging in takes 10 plus seconds.... I checked WHM to confirm CPU issue was occurring and it was at 5%, so I commented out the line suggested above and that in fact solved the problem, similar to what occurs when the privacy setting is manually changed to "anyone can see blog...." ....

    So... as best as I can tell it definitely seems to be some sort of redirect loop scenario.. in any case, we seem to be an isolated case, so we'll just be sure to have line 68 commented out in the future. Really appreciate the help and feedback we got here, its always nice to get to the bottom of some weird behavior and get it sorted out. Thanks!