Defender gets stuck on File Comparison

The Scan Report shows several core files that appear to be modified. When I click on any file listed the plugin is supposed to show a File Comparison but the progress bar is stuck with just a single pixel of progress and it never progresses or displays the comparison. All the files shown are harmless (log files and license.txt) so I don't believe they are the cause of the problem, but simply exhibiting the symptom of an underlying problem.

  • Predrag Dubajic

    Hi Steven,

    Thanks for starting new thread for this issue, could you please grant support access as well so we can have a closer look at this.

    If you missed the part about this in your other thread here's instructions link for granting access:
    http://premium.wpmudev.org/manuals/wpmu-dev-dashboard-enabling-staff-login/

    Please respond in this thread once access is granted.

    Best regards,
    Predrag

  • Predrag Dubajic

    Hi Steven,

    Thanks for granting access, I had a look at your site and the line you're seeing is not a loader, it's actually code wrapper but for some reason it doesn't load any code for you.

    The second strange thing I noticed is wp-config-sample.php in your results, this file should actually be ignored in defender scan by default, and that's how it works on my end, but for some reason on your site it does get included in results.

    I had a chat with plugin developer about this and he also didn't see these issues before so he would like to investigate it further if you could provide us with FTP or cPanel access.

    You can send us your details using our contact form https://premium.wpmudev.org/contact/ and the template below:

    IMPORTANT: Make sure you select "I have a different question" for your topic so it doesn't go back to forums - this and the subject line ensure that it gets assigned to me.

    Subject: "Attn: Predrag Dubajic"
    - Site login url
    - WordPress admin username
    - WordPress admin password
    - FTP credentials (host/username/password)
    - cPanel credentials (host/username/password)
    - Link back to this thread for reference
    - Any other relevant urls

    Best regards,
    Predrag

  • Steven

    I may have found the cause. I did a Network Deactivate and Delete then tried to Install again but could not get it to install... hmm. something else is going on.

    I recently switched to PHP 7 (tried 7.1.0 and 7.0.14 as both are supported by SiteGround) for the performance improvements. On a hunch, I reverted back to PHP 5.6.29 and tried installing again. This time it installs as expected. The scan still hangs and The Hub does not thing Defender is installed ... but at this point I am going to assume my configuration is scrambled so I am going to wipe and fresh install. I will give you an update once back online.

  • Steven

    It's stuck doing checksum compares again... Scan has been running for two days straight. Progress bar got to 100% and stuck there for a long time, then timeout error and it looks like it's stopped, but it hasn't. Looking at the active cron jobs shows that it's slamming calls to wd_check_checksum as fast as it can with no end in sight. I went into configure screen and disabled all 3 scan types but that didn't change anything. Trying to Run New Scan just tells me a scan is already in progress. Disable the plugin does stop the cron spawning, but as soon as I enable it again it immediately resumes spamming the con jobs for wd_check_checksum again...

    I am ready to throw the whole thing out... WPMUDEV as a suite of supported and trusted plugins is a great concept, but damn if I haven't spend 30+ hours in the past week chasing this and Hummingbird. I need to spend my time on my business and content, not troubleshooting the tech.

    Please let me know if you can find anything...

    BTW: The site is running PHP 7.0.14 since disabling the Live Feed plugin that had problems with it.

  • Steven

    FYI - When you do login. The WP Crontrol plugin is installed so you can see in each site what is scheduled to run under Tools / Cron Events. This screen will also confirm that DISABLE_WP_CRON is set to true in wp-config.php but there is a true cron job scheduled that fires off the wp-cron.php for each site every minute. I also modified the wp-cron.php to output some status info that shows up in the email I receive after every run, so I can confirm that these jobs are indeed being run and consumed. Here is a copy of the last run, which is pretty typical:

    =====
    05:06:02 Processing 3 sites, 13333333.333333 microseconds between cron jobs
    05:06:02 #1 = curl https://www.edsvcs.com/wp-cron.php
    ---05:06:03 (recurring) [postindexer_firstpass_cron] --05:06:03
    ---05:06:03 (one time) [wd_check_checksum] --05:06:03
    ---05:06:03 (recurring) [postindexer_secondpass_cron] --05:06:03
    ===
    05:06:16 #2 = curl https://www.edsvcs.com/edfocus/wp-cron.php
    ---05:06:18 (recurring) [postindexer_secondpass_cron] --05:06:18
    ---05:06:18 (one time) [wd_check_checksum] --05:06:18
    ===
    05:06:31 #3 = curl https://www.edsvcs.com/genedacademy/wp-cron.php
    ---05:06:32 (recurring) [postindexer_secondpass_cron] --05:06:32
    ---05:06:32 (recurring) [postindexer_firstpass_cron] --05:06:32
    ---05:06:32 (one time) [wd_check_checksum] --05:06:32
    ===

  • Steven

    So... after a bunch of troubleshooting and researching and reinstalling the site from scratch with default theme it seemed like I was onto something.. Several things actually ...

    1 - The transient locks used by the cache module were not sticking - that is to say that all transient values were being returned blank, which caused the wp-cron.php to continually bail out on direct calls to wp-cron.php?doing_wp_cron=<value> after failing the check on line 88. Calling wp-cron.php directly would run a job because line 77 sets both $doing_cron_transient and $doing_wp_cron to the same value right before the check on line 88, but then the call to _get_cron_lock() on line 120 would come back blank, causing the routine to think another process had stolen the lock so it would bail out.

    2 - Even after disabling and uninstalling Defender, once reinstalled, it would respond to any request to begin a new scan by reporting that it is already performing a scan. If I wait long enough, the scan eventually appears to abort more than a day later with a message something like the scan time exceeded 1500 seconds (should have snagged a screen capture... didn't). Clicking to start a new scan just resumes the prior scan with the progress bar already at 100% but an endless stream of cron jobs calling wd_check_checksum...

    3 - Once the scan finally completed, the results included 281 duplicate entries about the same unrecognized file ... causing me to wonder if it scanned the entire directory structure 281 times or just got hung up on that one file for some reason? (the file is one that I added during troubleshooting so a system scheduled cron job can fire wp-cron.php for every site in the network... the problem existed before I created the file).

    During the process of troubleshooting, SiteGround tech support helped me setup a cron job to run wp-cron-multisite.php once every minute. We also set "define('DISABLE_WP_CRON', true);" in wp-config.php. I could see that the job was consuming cron jobs but still Defender ran and ran for another 24 hours without completing.... During the process of setting this up, we found that using wget to call https://(mysite.com)/wp-cron.php did not work because SuperCacher would service the request and never call the php file. I added this file to the SuperCacher exclusion list and we switched to curl instead of wget. Between those two changes we got past the cache and I could confirm the php file was executing and consuming cron jobs.

    I created an entirely new account on SiteGround and installed WP MU but with just one site. The only plugins I installed were from SiteGround (SG CachePress) and WPMU DEV (Dashboard, Defender, Hummingbird, Smush Pro, Domain Mapping, Snapshot, Google Analytics+, Avatars, SmartCrawl, Post Indexer, and Comment Indexer). Once I told Defender to scan it did the same thing ... progress bar quickly went to 100% and stuck there. I loaded the WP Crontrol plugin and sure enough could see that cron jobs were queued up but not being processed...

    After reading more I found that WP will manage transients differently if Memcached is enabled, so I disabled SiteGround's SuperCacher including Level 1, Level 2, and Memcached. I did this on both sites - the original site we have been troubleshooting and the new site. I also restored WP's default cron handling "define('DISABLE_WP_CRON', false);" and suspended my manual cron jobs. About 10 minutes later both sites finished scanning. On the original site Defender reported 263 warnings for an unrecognized file (wp-cron-multisite.php) and the new site it showed 281 warnings about the same file.

    All this led me to think the whole thing must have been caused by Memcached and SuperCacher.... although I had no idea why it suddenly became a problem on Dec 24 when it had been fine until then. However, this theory seems to be blown up by further testing because I have re-enabled SuperCacher (Level 1, Level 2, and Memcached) and have been able to successfully run Defender scans on both sites several times without issue...

    Any ideas? Does any of the above make sense or look familiar to anyone? I have spent way too much time on this but I am reluctant to invest a lot of time into site content without confidence the underlying environment is stable.

  • Steven

    WP default cron handling is queueing up crons but not running any. In the past I found this was because the check on line 88 of wp-cron.php fails because doing_cron_transient is empty...

    My system scheduled cron job is running every 5 minutes and is consuming a single job each time before aborting because the check for _get_cron_lock() on line 120 returns an empty value for the doing_cron transient...

    I am not sure when it stopped working... if it was when I disabled caching as mentioned above or when your guys were in the system (they left Debug Bar and Debug Bar Cron plugins behind... those were not installed when I went to bed last night...).

  • Steven

    I am convinced the root of this problem lies in WP use of transients and interaction with the SiteGround Supercacher Memcached. Since I had previously simply disabled the plugin and used cPanel to disable the cache, I wondered if WP was still trying to use it. I re-enabled the plugin and sure enough it's config screen showed Memcached active even though it was disabled on the server. I could not disable it through the plugin until I re-enabled it through cPanel, at which point I could disable in the plugin and then disable again in cPanel. At that point cron jobs began processing again and immediately cleared the backlog.

    I think we can safely conclude that the issue with cron jobs not processing is one for SiteGround so I will report it to them.

    I don't know if this resolves the initial issue here of Defender getting stuck and processing the same files hundreds of times, but at this point let's close this ticket and I will open a new one if the problem occurs again.

    • Steven

      Transients are always used, it's just a question of which method. With memcached active they are not persisted in the database. In fact, this is one of the ways I was able to determine that WP still thought memcached was active even after it had been disabled. It turns out the SG SuperCacher plugin forces the setting and does not clear it even if disabled or the plugin deleted, and you can't clear it unless you first actually enable it, and then it will allow you to disable it again.