New to the community? Start here

"Service is currently under maintenance"

using wp 3.2.1, network child theme....having loads of problems :slight_frown: Currently, I cannot access my site on the front end or back end. I am receiving this message: "Service is currently under maintenance". I am trying to get back into my dashboard so I can attempt to resolve why buddypress and is breaking my site. TIA,
Kel

  • ClvrTv

    Interesting! I know this thread was from a while back but 1-2 months ago we also started seeing this intermittently too. It can happen on any site in our network and a single refresh usually clears it but it's kind of annoying. Any more insight on this?

    @Kel did you resolve?

    Seeing this in the logs that I think is related:

    [Thu Dec 22 14:40:43 2011] [warn] [client xxx.xxx.xxx.xxx] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server
    [Thu Dec 22 14:40:43 2011] [error] [client xxx.xxx.xxx.xxx] Premature end of script headers: index.php

    Might switch to suPHP instead of FastCGI but I'd really like to understand the issue more. Was interesting to me that this thread was the first hit on Google for "service is currently under maintenance" incl. quotes :slight_smile: Nice job WPMUdev!!!

    Anyway anyone have any nuggets of wisdom here? Again, didn't even see this issue until a month or two ago. Happens maybe a handful of times per week but the server self-corrects. At first I thought we were burning out our max_connections in mySQL but bumped those up and still see the issue. Fairly high-traffic multisite network.

    Would love to know if anyone has seen / understands this issue. Many thanks!

  • ClvrTv

    Um... this is the first time I've had to dive into this area, so I'm still figuring things out, but how are you using DSO and still allowing users to upload files? Not to mention any files that plugins need to create etc, since DSO runs PHP as the 'nobody' user and will result in permissions issues. Would love to hear any specifics you can provide about the "lots of tweaks" part :slight_smile:

    Also, I am wondering if it is something related to or deeper than mod_fcgid because even under suPHP still seeing an occasional similar issue although the error apache passes is slightly different:

    Service is temporary unavailable

    (Yes, grammatical awkwardness and all)

    Hopefully one of the server experts has a nugget :slight_smile:

  • Mark de Scande

    Here is my trick :slight_smile:

    How to set up server to run DSO

    First of Fire Wall + ModSecurity
    http://www.configserver.com/cp/csf.html
    http://www.configserver.com/cp/cmc.html
    http://www.configserver.com/cp/cmq.html

    Now run WHM EasyApache (Apache Update) and pick what you need i can not tell you what to use but i will make some point

    Apache 2.2 + PHP 5 / 5.3.8 + EAccelerator for PHP + Mod Security + Suhosin for PHP + Xcache for PHP

    Now there is a lot of stuff you can remove from Exhaustive Options List have a look around but one thing i whould advise is to add any thing to do with Cache / Deflate

    Note on Deflate Just add there lines to your .htaccess

    <IfModule mod_deflate.c>
 AddOutputFilterByType DEFLATE text/html text/xml text/css text/plain
 AddOutputFilterByType DEFLATE image/svg+xml application/xhtml+xml application/xml
 AddOutputFilterByType DEFLATE application/rdf+xml application/rss+xml application/atom+xml
 AddOutputFilterByType DEFLATE text/javascript application/javascript application/x-javascript
 AddOutputFilterByType DEFLATE application/x-font-ttf application/x-font-otf
 AddOutputFilterByType DEFLATE font/truetype font/opentype
</IfModule>

    Now click Build it take about 1hour max

    It will give you a pop up it will tell you select PHP look for
    PHP 5 Handler dso
    Apache suEXEC off

    This is the important part
    Use Putty log in to your server run the
    Change blogline to your account name and run :

    chmod -R 777 /home/blogline/public_html/
    chmod -R 777 /home/blogline/public_html/wp-content
    chown -R blogline:blogline /home/blogline/public_html/

    Tweek PHP
    memory_limit 98
    upload_max_filesize16M
    max_execution_time 140
    max_input_time 150

    Tweek Apache Configuration >> Global Configuration
    Start Servers 15
    Minimum Spare Servers 20
    Maximum Spare Servers 20
    Server Limit 650
    Max Clients 650
    Max Requests Per Child 25
    Keep-Alive On
    Keep-Alive Timeout 2
    Max Keep-Alive Requests 10
    Timeout 15

    Remember we run a dedicated server with 16Gb of ram and 8 Cores

    Now my best trick i have is go to your fire wall and look for Connection Tracking change it to 50

    As for MySQL

    SFTP in to your BOX /etc look for my.cnfand edit it
    I have pasted mine

    #DO NOT MODIFY THE FOLLOWING COMMENTED LINES!
    #Created with ELS from http://www.servermonkeys.com
    #els-build=5.0
    [mysqld]
    local-infile=0
    datadir=/var/lib/mysql
    skip-locking
    #skip-innodb
    skip-networking
    safe-show-database
    query_cache_limit=1M
    query_cache_size=128M ## 32MB for every 1GB of RAM
    query_cache_type=1
    max_user_connections=200
    max_connections=500
    interactive_timeout=10
    wait_timeout=20
    connect_timeout=20
    thread_cache_size=128
    key_buffer=1024M ## 128MB for every 1GB of RAM
    join_buffer=1M
    max_connect_errors=20
    max_allowed_packet=16M
    table_cache=1024
    record_buffer=1M
    sort_buffer_size=8M ## 1MB for every 1GB of RAM
    read_buffer_size=8M ## 1MB for every 1GB of RAM
    read_rnd_buffer_size=8M ## 1MB for every 1GB of RAM
    thread_concurrency=16 ## Number of CPUs x 2
    myisam_sort_buffer_size=64M
    server-id=1

    [mysql.server]
    user=mysql

    [safe_mysqld]
    err-log=/var/log/mysqld.log
    pid-file=/var/lib/mysql/mysql.pid
    nice=-5
    open_files_limit=8192

    [mysqldump]
    quick
    max_allowed_packet=16M

    [mysql]
    no-auto-rehash
    #safe-updates

    [isamchk]
    key_buffer=64M
    sort_buffer=64M
    read_buffer=16M
    write_buffer=16M

    [myisamchk]
    key_buffer=64M
    sort_buffer=64M
    read_buffer=16M
    write_buffer=16M

    [mysqlhotcopy]
    interactive-timeout

    I hope this helps some one out there with there server set up :slight_smile: +1

  • ClvrTv

    @Bloglines thanks for the offer! Unfortunately I can't take you up on it as Clvr.Tv is a commercial network with business clients and our sysadmins are pretty protective. I had them evaluate the DSO option and there are particulars of our environment that prevent us from going down that route. But I'm glad it's working well for you!

    Due to a bug in suPHP that was getting tripped by InifiniteSEO we've gone back to fastCGI. It's ok for the most part. But there is a certain set of conditions that seem to trigger the bug there, and a particular plugin seems to be able to predictably trigger it perhaps with heavy database queries in a short amount of time (like on a large e-commerce cart product inventory import).

    Anyway, still collecting data on it, but seems to be livable for now. We also had a disk going bad in our RAID array which I think exacerbated the issue. So that's fixed now.

    Would still love to know if anyone else has any other insight / experience on this issue and what specifically might trigger it.

  • ClvrTv

    Ok, the plot thickens, and maybe a break-through (?) but only time will tell. However, the latest development is an excellent opportunity to provide some new focus to this thread that I think will have a lot of value for the entire multisite community. It's probably already Google-able too somewhere out there, but would love to hear from the experts here:

    Our host has made the observation that fastCGI has a default setting / limit of 10 daemons per site folder. Thinking maybe this was the issue they have increased that for us. Naturally, WP Multisite runs from a single codebase / "site" / like under /public_html or whatever. Could this indeed be the culprit? On the one hand it seems like yes, but on the other hand I have doubts, because technically Multisite is still a single site - it is just serving multiple "sites" from the database. However, when using the Domain Mapping plugin, does this then require multiple simultaneous daemons from fastCGI ? And could this maybe be the fastCGI bug??? Not really a bug but a configuration issue?

    That begs the question that I'd really love to have the WPMUdev server experts weigh in on: What other WordPress Multisite specific server configs / optimizations should be part of any large-ish multisite installation? Obviously, there are multi-server / multi-db setups possible, but this is still single-server context.

    Ok, I will stop spamming now I promise. :slight_smile: Although I might chime again with a status if anything becomes super clear as to whether this fixed our issue or not.

  • Mustafa

    Hi everyone,
    Actually I can't understand problem.server based or software (themes,plugins etc..)

    I tried fastcgi and suphp currently I'm using suphp because I get max upload problem with fastcgi.

    Can you give me more detail about your server configuration.Maybe you need managed level SLA or better hosting company.

    I'm writing some address in here - may it can help you.

    http://codex.wordpress.org/How_to_Scale_WPMU
    http://c3mdigital.com/2011/07/22/wordpress-performance-server/3/
    http://www.arnebrachhold.de/2007/02/16/four-plus-one-ways-to-speed-up-the-performance-of-wordpress-with-caching/
    https://premium.wpmudev.org/blog/scaling-wordpress-wpmu-buddypress-like-edublogs/

  • ClvrTv

    Thanks @Mustafa - I will go through those links!

    Everything has been more stable for a few days since since the fastCGI config chages. Here is the description of the change: "We've set fastcgi-per-class limit to the same value as max-fastcgi-process-number so basically per-class limit does not affect anything now. Max-fastcgi-process number is set to a reasonable value to avoid out-of-memory error and overloads."

    However, we did have another 20 minute hiccup today which was attributed to too many PHP processes, but no clues as to whether they were caused by traffic / load or rogue plugin / bad code. We've turned on slow query logging.

    Will post any other discoveries along the way.

  • ClvrTv

    Thanks for the follow-up reminder, Timothy! So much has happened and I think the dust is finally settling.

    Here is a summary of the entire saga :slight_smile:
    1) was running out of mysql connections (default was 50) bumped that up
    2) saw errors related to FastCGI switched to suPHP
    3) had issue with suPHP giving false positives for safe_mode, I described here:
    https://premium.wpmudev.org/forums/topic/set-time-limit-in-safe-mode-error
    4) switched back to FastCGI and found daemon per folder limit issue
    5) increased max-fastcgi-process-number as high as reasonable for 4GB RAM
    6) set fastcgi-per-class = max-fastcgi-process-number to nullify daemon per folder issue
    7) still saw snowball scenarios where PHP processes / mysql connections weren't closing and growing until crash / service restarts from apache health monitor
    8) added additional 8GB RAM for 12GB total - this was a stop-gap and understood to be a band-aid, but will help us with traffic spikes anyway
    9) FINALLY found a plugin that seemed like it was generating too many hanging queries that were accumulating and causing the snowball. Not convinced that was the true culprit, but it was activated network wide and I really only needed it on one site, so I deactivated it, updated it to the latest version :slight_smile:, and reactivated it on one site only.
    10) re-tweaked ALL applicable settings as optimal for 12GB RAM

    Stable and smooth since then. No more hung queries in the logs / process monitor.

    Somewhere in there I think we also bumped up a few other settings.... max timeout and a couple others, but can't remember now. I also audited every plugin and cleaned some of those out as well as finally re-implemented WP Supercache as Donncha was gracious enough to provide a quick supercache plugin based on my input to make it usable for Multisite with basic site-level caching on/off control, which was the whole reason I abandoned it a while ago:
    http://wordpress.org/support/topic/cache-plugin-for-multisite-how-to-not-cache-all-sites-in-network

    SO, the moral of the story (IMHO): FastCGI issues and other PHP handler-related issues (though there probably certainly are bugs) are likely related to bad plugin behavior or combinations of plugins or any of the other joyful complexities that the Multisite universe contains. :slight_smile:

    Troubleshooting this kind of think is like dropping a diamond in the middle of a game of beach volleyball and trying to find it after. LOL. Love it. Cheers.

  • Timothy Bowers

    Ouch, does in deed sound like a saga.

    12GB ram, wow, how many sites and users again?

    Stable and smooth since then. No more hung queries in the logs / process monitor.

    So this plugin, was it not closing queries correctly?

    Did you let the developer know as this could still cause more issues for you later if that site becomes busy.

    I've always found Donncha a great person to be honest! I have much respect for him. :slight_smile:

    Troubleshooting this kind of think is like dropping a diamond in the middle of a game of beach volleyball and trying to find it after. LOL. Love it. Cheers.

    Sadly server environments can be vastly different with many considerations. I never had issue with suPHP but thats probably how we have our set ups.

    Please you got this one sorted though, hopefully you won't have these issues again!

    Thanks for the update on this one, it may surely help other users facing similar issues! :slight_smile:

  • ClvrTv

    12GB ram, wow, how many sites and users again?

    Yes, overkill assuming everything else is optimized correctly :slight_smile: 300 sites, roughly as many users. But we have a unique network. Most sites are full-blown business web sites, many of them custom, with roughly 30 mu-plugins and network activated with another 50 plugins installed and activated in various combinations across the sites. (That's after my clean-up scourge :slight_smile:

    So this plugin, was it not closing queries correctly?
    Did you let the developer know as this could still cause more issues for you later if that site becomes busy.

    That's the thing. I'm not convinced that our host / sysadmins had that identified correctly. It had update queries that were hanging and accumulating during backup locks and then not clearing after backups ran. In any case, deactivating network wide and updating to the latest version seemed to fix that particular symptom. Sysadmins made it out to be more than that, but that was the only concrete evidence I could get. Maybe dev fixed that issue in an update. Never had bandwidth to go back and figure out 100%

    Thanks for the update on this one, it may surely help other users facing similar issues! :slight_smile:

    Sure thing! I know I've benefited immensely from the WP community at large, so hopefully this does give someone some ideas if google brings them here on their quest for similar solutions.

Thank NAME, for their help.

1 Point
5 Points
15 Points
Other

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.