Multi-Domain Nginx tip?

Hi Guys,
So I did some traffic storm test on our site and apache died around 85 users with CPU at 95%..
So I am trying to get nginx installed to hopefully improve performance.

Can someone volunteer their configuration file with multi-domain, Any tips..
Thanks
Joe

  • jozwikjp

    That sounds like a wise tip :slight_smile:
    So I have it installed now. Just waiting for my host to update our load balancers to point the traffic properly.

    But I have it configured and php is loading.. Won't know if I got my url rewrites correct until they update the firewall. Finger are crossed.

    I found some good instructions for my centos 5.2 install.

    http://library.linode.com/web-servers/nginx/php-fastcgi/centos-5

    I had to update my kernel to

    2.6.18-194.26.1.el5
    with "yum update kernel kernel-devel"

  • jozwikjp

    Sweet.. seems to be working aok with multi site and multi domain. :slight_smile:
    I tweaked my config a slightly from the ones I could find on the forums here and other places.

    My Setup

    loadbalancer port 80
    |
    nginx server port:83

    On one server I upgraded php to 5.3.3 as I read that php-fpm setup is faster. That hasn't gone well so far and we used some compiled scripts using nucoder and ioncube. And I can't those to work yet..

    But my working setup..
    php 5.1.6 spawn-fcgi
    nginx 0.8 something..

    here is my nginx configuration file.

    server {
    listen 83;
    # listen 81;

    # listen somename:8080;
    server_name _;
    server_name_in_redirect off;
    port_in_redirect off;
    location ~ \.php$ {
    # root html;
    fastcgi_pass localhost:9000;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /home/msi/htdocs$fastcgi_script_name;
    include /etc/nginx/fastcgi_params;
    #include fastcgi_params;
    }

    location / {
    root /home/msi/htdocs;
    index index.php;
    gzip on;
    gzip_http_version 1.0;
    gzip_vary on;
    gzip_comp_level 3;
    gzip_proxied any;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
    gzip_buffers 16 8k;
    if (!-e $request_filename) {
    rewrite ^.+/?(/ms-.*) $1 last;
    rewrite ^/files/(.+) /wp-includes/ms-files.php?file=$1 last;
    rewrite ^.+/?(/wp-.*) $1 last;
    rewrite ^.+/?(/.*\.php)$ $1 last;
    rewrite ^(.+)$ /index.php?q=$1 last;
    expires 10d;
    break;
    }
    }
    location ~* ^.+.(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js)$
    {
    root /home/mysiteinc.net/htdocs;
    rewrite ^/.*(/wp-.*/.*.(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js))$ $1 last;
    rewrite ^.*/files/(.*(html|jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js))$
    /wp-includes/ms-files.php?file=$1 last;
    if (!-e $request_filename) {
    rewrite ^.+/?(/wp-.*) $1 last;
    rewrite ^.+/?(/.*.php)$ $1 last;
    rewrite ^(.+)$ /index.php?q=$1 last;
    }
    expires 30d;
    break;
    }

    }

  • jozwikjp

    Installed varnish - pretty sweet.
    And this plugin..
    https://github.com/pkhamre/wp-varnish/blob/master/README.markdown
    with nginx as the front..
    so varnish - nginx , php-fpm, apc
    varnish 2.1.3

    1 server
    500 concurrent users
    1 - 5 % cpu
    Very fast site..

    top - 18:44:52 up 2 days, 8:03, 2 users, load average: 0.33, 0.28, 0.27
    Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie
    Cpu(s): 0.8%us,

    I am pretty surprised at how much of the site seems to work as if it wasn't cached.
    Site preview on templates doesn't work. But this is pretty remarkable.

    20 minute test
    Response
    (average) Errors Requests RPS
    (average) RPS
    (peak) Throughput
    (average) Throughput
    (peak) Total
    Transfer
    HTML 0.325 32,439 112,838 90 124 1,209 kB/s 1,658 kB/s 1 GB
    Other * 0.378 3 225,553 179 260 3,875 kB/s 5,582 kB/s 5 GB
    Total 0.360 32,442 338,391 269 380 5,084 kB/s 7,178 kB/s 6 GB

  • jozwikjp

    My total setup is

    [Haproxy lb:80] - > [web1 varnish:83] ->[web1 nginx:82]

    Before I was using the HTTP_X_FORWARDED_FOR to set the IP for the clients..

    I don't think that is going to work anymore. Or at least I haven't been able to get it yet. The closest I get is the IP of [Haproxy lb].

    All the domains on my network seem to load very fast.
    -The buddy press admin bar doesn't seem to work correctly. When you are logged in wp-admin the admin bar responds as you would expect. When you are just on a webpage it appears as though you are still logged out.
    -The affiliate plugin (setting the cookie) seems to work aok.
    -preview a post doesn't seem to work.. throwing a "you don't have permission error".
    -posting and updating posts goes live right away.
    -preview themes just shows the current theme.
    -user registration works
    -IP crap makes anti-splog not work so good. So I had to disable that as most IP's are 127.0.0.1
    -I would imagine ad sharing doesn't work so good or at least the same.
    I am guessing the affiliate plugin will record the single hit per user just the first time they load.. That might be how it already works though.

    I feel significantly relieved though, as I was getting really concerned of the money we were going to have to dump on extra database servers and web servers to grow.

    turning on mysql db cache at the database level helps alot with wordpress. Most all the queries when the server was getting slammed were being delivered from cache.

    I think I am going to increase the query cache..
    query_cache_size = 64M
    query_cache_limit = 2M

  • jozwikjp

    Thought I would give a update on this.
    These are my updated rules for my varnish vcl file. This fixes the buddypress admin bar so you appear logged in besides just the admin bar.
    Also the site preview work now as well.

    Load storm test right now is at 250 users..
    CPU usage is at 1% so thats cool.

    backend default {
    .host = "127.0.0.1";
    .port = "82";
    }

    acl purge {
    "localhost";
    }

    sub vcl_recv {

    if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
    # No point in compressing these
    remove req.http.Accept-Encoding;
    } else if (req.http.Accept-Encoding ~ "gzip") {
    set req.http.Accept-Encoding = "gzip";
    } else if (req.http.Accept-Encoding ~ "deflate") {
    set req.http.Accept-Encoding = "deflate";
    } else {
    # unknown algorithm
    remove req.http.Accept-Encoding;
    }

    if (req.request == "PURGE") {
    if(!client.ip ~ purge) {
    error 405 "Not allowed.";
    }

    purge("req.url ~ ^" req.url "$ && req.http.host == "req.http.host);
    }

    if (req.request != "GET" &&
    req.request != "HEAD" &&
    req.request != "PUT" &&
    req.request != "POST" &&
    req.request != "TRACE" &&
    req.request != "OPTIONS" &&
    req.request != "DELETE") {
    return (pipe);
    }

    if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
    }

    if (req.url ~ "wp-(login|admin)") {
    return (pass);
    }
    if( req.url ~ "^/wp-content/themes/blogs-mu/global-adminbar.php" || req.http.Cookie ~ "wordpress_logged_in_" ) {
    return (pass);
    }
    if( req.url ~ "^/?preview" || req.http.Cookie ~ "wordpress_logged_in_" ) {
    return (pass);
    }

    }

    remove req.http.cookie;
    return (lookup);
    }

    sub vcl_fetch {
    if (req.url ~ "wp-(login|admin)") {
    return (pass);
    }

    set beresp.ttl = 24h;
    return (deliver);
    }

  • Aaron

    I ran into that symptom myself, though using spawn-fcgi. Finally tracked it down to APC. It seemed to be causing occassional segfaults in php-cgi. Also sometimes php would return only part of the php page, cutting off halfway.

    Never figured out why the segfaults, chocked it up to some incompatibility with php-cgi and APC. Same thing happened though with eAccellerator/xcache when I tried those instead. Eventually just disabled all accelerators, and putting up with the slowness.

    When I find time I'm going to recompile everything again with the latest versions, especially of PHP-CGI/APC. Perhaps my versions were somehow incompatible. The most annoying thing about nginx is lack of PHP error details in the log. I'm sure there must be some way to set it up, but still havn't found it.

  • jozwikjp

    Ok so I looked at my error log for php-fpm.. And it looks like where it dies is..

    WordPress database error Got a packet bigger than 'max_allowed_packet' bytes for query UPDATE wp_options SET option_value = 'a:4663:{s:53:\"b2c80bfa934

    But at that point this goes on forever.
    This update query just goes and goes..
    at the end of the query
    ' WHERE option_name = 'cross_domain' made by require_once, do_action, call_user_func_array, multi_domain->build_cookie, multi_domain->build_blog_cookie, update_blog_option, update_option, m_wpdb->query
    [06-Dec-2010 05:22:54] PHP Notice: Undefined index: debug in /home/mysiteinc.net/htdocs/wp-config.php on line 103
    [06-Dec-2010 05:22:54] PHP Deprecated: Function set_magic_quotes_runtime() is deprecated in /home/mysiteinc.net/htdocs/wp-settings.php on line 27

    Any tips?