Lost settings again, recharging for accounts & withdrawing pro status

Hi,

I've had these problems numerous times now and I'd love to get to the bottom of them as it's affecting the business of our site.

The entire Pro Sites network settings have just vanished again, except the account levels and upload quotas, as if they've corrupted themselves and reset to default. All plugins and themes that I'd set up, PayPal details and prices are gone and it takes me over an hour each time this happens to re-set everything up again. Disabling all plugins to find the culprit is not an option as we have thousands of customers, I can see no reason why anything we would have installed would mess with the settings. I'll list everything we currently use:

http://c.ed.gs/JWAh

* All these plugins are network activated, we also use Multi-DB.

Server set up is Ubuntu 10.04 LTE, nginx, php-fpm with APC, MySQL, no object-cache, no WordPress security plugins or plugins that change core features randomly and WordPress 3.4.2.

I'd previously been restoring the psts_settings table in the wp_sitemeta database, but this time I don't have a recent back up. It hasn't actually done this for a while now, nothing has changed since last night to make this happen.

Also, I had a customer the other day who had had their Pro account status withdrawn automatically, and then sent a receipt and charged again by PayPal, even though they had 6 months left on their account. I tried to refund the PayPal payment via the plugin but it just said that it had found an error, that was all. Account statuses are withdrawn from people regularly at random by the plugin, or culprit of the problem, but this was the first time I've seen it charge again. This, to me, is quite worrying.

Do with that what you will, I'm more than happy to work with anyone that will help rectify this. It is a major problem and needs fixing immediately, although I know no one else seems to have the same problem as me, or at least hasn't noticed it yet.

Cheers,

Ed

  • Aaron

    As with you last report I really don't know what to do for you. We can't reproduce it, and noone else has reported it. We use pro sites with multidb on edublogs.org, and never ran into your issue.

    So we know it's something specific to your install, and I have no ideas on how to debug really. My best guess is it's tied to the serialized settings array getting corrupted. I have seen this with multidb, and it was tied to the encoding of the database tables. I'll ask our whole dev team if they have any ideas at all.

    Still I recommend keeping backups of your settings as you said:

    I'd previously been restoring the psts_settings table in the wp_sitemeta database,

    The settings don't really change unless you do it, so age isn't so important.

    As far as:

    Account statuses are withdrawn from people regularly at random by the plugin, or culprit of the problem, but this was the first time I've seen it charge again. This, to me, is quite worrying.

    Pretty sure that would be something completely unrelated, lets start debugging it. First off as far as when charges are made, that's completely controlled by paypal based on the dates and info passed to it at initial checkout. As far as status being withdrawn, that is only triggered by:
    - The expire field being passed (this is what is extended by payments or manually). Sometimes this is triggered normally if the next sub payment is delayed a day or more. Sometimes it's just for a few hours if things align just right as Paypal processes scheduled payments in the middle of the night, not necessarily exactly 1 month/year to the hour from the last payment or checkout.
    - Manual revoking of access via the admin
    - An IPN notification from paypal like for a chargeback, refund, payment deny, etc.

    withdrawn automatically, and then sent a receipt and charged again by PayPal,

    This actually sounds like normal behavior for a scheduled payment. It would help alot if you could paste their entire action log here.

  • PrimaryT

    First off, you can ignore the PayPal query. The user was hasty with their complaint, without checking if PayPal had actually taken money from them again, it had just sent the receipt email via Pro Sites again.

    Yeah, it's as if when adding a new plugin, theme or setting it can corrupt the serialisation, I witnessed this happen manually when re-adding plugins and themes to the psts_settings row, could this be the same for the blogs being withdrawn or are they not serialised?

    The old backup had a fair few plugins and themes missing from the list so I had to go through manually to check the missing new ones.

    Thanks for the reply Aaron.

  • PC

    Hiya,

    Greetings and thanks for being a great community member.

    We haven't heard from you on this one for long and I am doing a regular followup to see if there is still something we can assist you on this thread.

    Just to manage the support issues more efficiently, I am marking this thread as resolved for now however this is not being done to avoid your questions in any ways.

    Please feel free to mark this is "Not resolved" in case you have further questions and we would be back on it.

    Thanks a lot for being with WPMU DEV.

    Cheers
    PC
    Sales &Support

    Did you know we offer FREE lifetime memberships? Click here to learn more.

  • PrimaryT

    It's happened again, it was due to updating plugins this time. I also can't reimport the SQL from before as it breaks serialisation again however I try and do it, so it's back to adding all 300 plugins again. I'm using a sub-network within my network too, that seems to have kept all of it's settings without error, although I'm not using premium plugins as an option. I don't have a list of the plugins I updated yet, I will update this thread later with them once I check what they were.

    Cheers,

    Ed

  • PrimaryT

    Right, here's the plugins I updated:

    BBPress
    BBPress Threaded Replies
    Co-authors Plus
    EWW Image Optimizer
    Faceboko
    Network Latest Posts
    Simple Page Ordering
    WP Email Login
    Yet Another Related Posts Plugin

    Plugin I installed roughly at the same time:

    WordPress SASS

    Themes Updated:

    Hatch
    Oxygen
    Pink Touch 2
    Responsive
    Simple Catch

    I have a sneaking suspicion that it's the yet-another-related-posts plugin, have a feeling that last time I updated it the same thing happened.

  • PrimaryT

    Ok, more info...

    It seems to happen if PHP is overloaded or crashes. I'm using PHP-FPM, PHP 5.4.8-1~lucid+1. It had added 197 extra defaulted serialised rows for one of the networks, I use WordPress Multi-Network and the other network never seems to have the same problems.

    I'll keep adding info to this thread if that's ok, I really need to get to the bottom of this, it's affecting our service.

    Cheers,

    Ed

  • PrimaryT

    Another update...

    The psts_levels row keeps corrupting itself daily now as well, it creates many new duplicated rows with the default data. Withdrawing accounts is also happening a lot more regularly, I've added a wp_mail call to the withdraw function in pro-sites.php so I can notified and re-add their status when it's removed. I've not managed to make any correlations between anything running and the corruption happening.

    Here's something else I've just discovered regarding PayPal subscriptions expiring after the first month, not taking the recurring subscription into consideration.

    I've commented the log from the Pro Sites Management area at http://c.ed.gs/LBix.

    The user assures me that payment has been taken from her monthly, but no log has been made in the Pro Sites Management area. Some photographic evidence below:

    http://c.ed.gs/LBpW & http://c.ed.gs/LBjj

    I then looked at other pro status sites and it appears that they all withdraw at the end of the month and they have to cancel their subscriptions and re-sign up each month, no one has told me of this apart from the ones that fail monthly.

    I've tried grepping through all plugins that are in use network wide and on the blogs in question for functions that may relate to one in use in pro-sites.php or other Pro Sites files, I can't find anything bar a similar withdraw function in BuddyPress, which we don't use at all.

    I really could do with some help debugging this, happy to try anything you can think of to test. This can't be the end of the issue, I understand you may be busy but this could happen to someone else eventually, so would be good to get it fixed now.

  • Aaron

    That would be a symptom of not entering the IPN url in your paypal account as you should. Payments are made, but paypal is not notifying pro sites so they get the withdraw.

    As far as the corrupting arrays, seems we know exactly what is happening but not why. Are there any other serialized array options in sitemeta you can watch to see if they are corrupting also?

    I had a thought, not sure if we discussed it before but if you are using some kind of object cache that could be causing this. Duplicate options or corrupt options from our experience can often be tied to object cache issues.

  • PrimaryT

    Many thanks for replying Aaron, really appreciate the help.

    It appears I have been an idiot and not read the IPN notifications bit as I thought it was for PayPal Pro, I'll get that sorted.

    There's quite a few rows in sitemeta obviously, some relating to other plugins we use, but none are corrupted or have done.

    No object caching, tried it several times without success so have always removed it.

    Cheers,

    Ed

  • Aaron

    I have a guess that this may be where the dupes come from:

    //update install script if necessary
    		if ($this->get_setting('version') != $this->version) {
    			$this->install();
    		}

    On loading the plugin, if it can't read it's stored version setting, then it will create default settings array, merging with any existing:

    $settings = wp_parse_args( (array)get_site_option('psts_settings'), $default_settings );
        update_site_option( 'psts_settings', $settings );

    That's normally a good thing though. The root is for some reason on occassion get_site_option('psts_settings') does not work for the entirety of the page load, causing core WP to create duplicat options.

    When it calls update_site_option, core WP is getting a hard false from get_site_option, telling it the option doesn't exist in the db and should be inserted fresh, creating a dupe with default settings.

    Again, the only possible thing I can imagine that would break get_site_option is an object cache plugin going wrong.

  • PrimaryT

    Main plugin configuration corruption has stopped due to me hacking the plugin after each update. Withdrawals are still plaguing us though, mainly for manually extended or bulk upgrades, but one or two subscriptions are still cutting themselves off if they're longer than a month. Still got nothing that sticks out as the cause of the problem. We're using mainly WPMUDev plugins, and I can't see any of those conflicting with it, nor nothing else we use should interfere either. I'm still not sure how it's checking the remaining time left on subscriptions, we previously thought that they may be getting corrupted some how but I've seen nothing in the database that would confirm this. But please, don't mark it as resolved, it really isn't. I'm having to hack the plugin myself to stop it creating problems.

    Cheers,
    Ed

  • Aaron

    DId you get the IPNs sorted?

    That would be a symptom of not entering the IPN url in your paypal account as you should. Payments are made, but paypal is not notifying pro sites so they get the withdraw.

    That was the extending problem you had. When a payment is made, paypal sends a notification to pro sites so it can extend. Are you seeing ipns in the logs of pro sites? You should be able to view 2-3 ipn log items in a sites log right after an initial pro sites checkout.

  • PrimaryT

    It happens to random accounts, just more noticeable on the longer running ones as they don't re-extend monthly. One month trial accounts are least noticeable, but a lot of them are doing it as well. Doesn't seem to be any correlation between anything, I've got a different sub-network that's started doing it as well on those sites as well, they're all permanently extended ones though, no subscriptions via PayPal.

    Cheers,
    Ed

  • PrimaryT

    I'm assuming it's a cron issue now. Now we're on one server, or PHP is only running on one server should I say, cron is only running once. I assume before that both servers could have been running the same cron job and clashing together to mess things up. I've no idea how you would run cron on a multisite install manually without taking the server down each time you ran it. How are other people achieving this on larger installs?

    Also the removal of Pro account on crash, I assume it was running a large cronjob of checking all expiration dates, but it ran out of MySQL connections and tanked out. Next thing I know, I've got a tonne of emails telling me they've all expired (due to my email hack). I still have one or two withdrawals, but it's only if I restart PHP, as if the cronjob doesn't complete fully.

    As a feature request, it would be useful to have withdrawal email notifications anyway so you could chase customers up and see why they didn't want to renew etc.

  • Aaron

    I've no idea how you would run cron on a multisite install manually without taking the server down each time you ran it. How are other people achieving this on larger installs?

    On edublogs we have some nginx rules to pass normal wp cron requests to their own php-fpm pool that has lower priority and timeouts.

    Also the removal of Pro account on crash, I assume it was running a large cronjob of checking all expiration dates, but it ran out of MySQL connections and tanked out.

    I'm not really at all sure how this is happening to you. There are no cron jobs or anything in pro sites. Basically what happens is every page load on a given site, the check() method is called:

    function check() {
    	  global $blog_id;
    		if ( is_pro_site($blog_id) ) {
    			do_action('psts_active');
    		} else {
    			do_action('psts_inactive');
    
    			//fire hooks on first encounter
    			if (get_option('psts_withdrawn') === '0')
    	      $this->withdraw($blog_id);
    		}
    	}

    So what it looks like is happening is is_pro_site() is returning false for you somehow, this is based on the 1 sql query on the pro sites table. So if this is somehow happening even when the expire in the table shows it's active, then the sql query is returning an incorrect value somehow. A db outage shouldn't affect this, it would cause an error rather than false for is_pro_site(). Are you sure you don't have a caching something in front of mysql?

    Now if the case was that the expire was passed, but it's just triggering the withdrawl() method then that might by tied to 'psts_withdrawn' blog option. But from your descriptions it's happening to current pro sites that are far from expire.

  • PrimaryT

    On edublogs we have some nginx rules to pass normal wp cron requests to their own php-fpm pool that has lower priority and timeouts.

    Interesting, I'll look into doing that.

    A db outage shouldn't affect this, it would cause an error rather than false for is_pro_site(). Are you sure you don't have a caching something in front of mysql?

    Caching wise, infront of MySQL we weren't using anything bar the inbuilt MySQL cache facilities. We're now using APC object-caching since yesterday, it has made no difference to anything as we're now quite stable and nothing has crashed or withdrawn unnecessarily. Next time/if it happens I'll check the logs and report back what the error was. I'm pretty sure our other server, the one we've taken out of the stack, crashed regularly and could have triggered whatever the issue is/was. Turns out the hard drive has a lot of bad sectors, so if it had bad tables that could have caused it maybe? Still when PHP crashed and ran out of MySQL connections on this server, which is all in working order, it did trigger it for nearly all pro site customers.

    Now if the case was that the expire was passed, but it's just triggering the withdrawl() method then that might by tied to 'psts_withdrawn' blog option. But from your descriptions it's happening to current pro sites that are far from expire.

    Most of the sites that were affected are either permanently extended or yearly/bulk upgrades.

    Cheers,
    Ed

  • PrimaryT

    It's just happened again on a couple of blogs when the server loaded up slightly. Here's the logs I got from nginx/php:

    2013/01/23 10:46:07 [error] 7408#0: *492500 FastCGI sent in stderr: "PHP message: WordPress database error  for query SELECT expire, level FROM wp_pro_sites WHERE blog_ID = '3458' made by require('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), do_action('init'), call_user_func_array, ProSites->check, is_pro_site, ProSites->is_pro_site, m_wpdb->query
    2013/01/23 10:46:07 [error] 7408#0: *492599 FastCGI sent in stderr: "PHP message: WordPress database error  for query SELECT expire, level FROM wp_pro_sites WHERE blog_ID = '13717' made by require_once('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), do_action('init'), call_user_func_array, ProSites->check, is_pro_site, ProSites->is_pro_site, m_wpdb->query

    Cheers,
    Ed

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.