[Defender Pro] Better decisions with centralized metrics

Defender Filescan commonly reports eval() functions in plugins. It would be helpful if it didn't report false positives, based on Ignore decisions by other Defender users. For example, plugin Foo gets flagged for an eval() at line 1030 column 4 for version 2.5.1. I choose to ignore that, and so does every other user of this plugin. So the next time someone runs a Filescan, and it detects this exact same condition, why should it warn the admin and ask them to make a decision when everyone else has already been through the investigation process and concluded that the flag is invalid.

Of course we can't really trust the crowd - what if way too many people simply hit that Ignore button, and it turns out the warning was good? The problem we need to avoid is where too many unqualified people hit that Ignore button. I may or may not be qualified to make a decision about the Foo plugin - do you know any better? In this case some help from our friends at WPMU DEV would help:

When more than X sites gets exactly the same warning, log it. Look into it. And somewhere in the Hub set a disable flag for "Foo/2.5.1/1030/4". Now we know someone has actually looked at it, and from that point forward no other site admin needs to see that warning. This can be a part of the Pro service offering, beyond the freeware offering.

Maybe that service doesn't need to come directly from WPMU DEV. Maybe we can make better decisions based on a factor of trust of others. In other words, if I'm on the development team for plugin Foo, and I know for a fact that that line 1030 is good, then my level of trust for that specific plugin, maybe 79 on a scale of 1 to 100, should be much higher than Joe Bloggs, with a rating of 2, who can't read PHP. All we need is a rating to be displayed in the File Scan page, like : "This is considered a non-issue by 82% of known users of this plugin with a credibility rating greater than 74".

How do we assign a factor of trust to people? Well let's stop here and consider that now we're trying to fine-tune a process that doesn't exist yet. A lot of questions could be asked about how to implement this. If the premise is valid we can create solutions for every challenge. If we aren't even doing this then we don't need to go down the rabbit hole to solve every possible problem. I'm saying we should start by considering a centralized process rather than the current paradigm where everyone is forced to make their own decisions for no good reason anymore given the technology available.

Consider also : When a site admin chooses to delete a PHP file, or an entire plugin, because it contains an eval, that's a flag to the rest of us that perhaps something is up. Does that person know something the rest of us should know? Wouldn't it be nice if Defender logs this significant event, or just clicks a new Warning button, so that someone at WPMU DEV can check to see if perhaps the rest of us should delete that specific Foo v2.5.1 plugin as well? Wouldn't it be nice if those of us with Foo v2.5.0 could be warned that v2.5.1 has some defect that should preclude our update to that release - and if Automate then refused to install that specific version without a manual override?

It's time to make use of the data that's available in these important plugins that have a connection back to the WPMU DEV servers. There are metrics there about how often sites trigger IP lockouts, about the class C IP blocks that are sourcing the most recent bot attacks, about how often there are unusual issues with specific plugins, about how often specific files keep re-appearing in folders (like favicon.ico) ... and lots of other things that are common to most of us.

Thanks for your consideration.

  • Tony G
    • Mr. LetsFixTheWorld

    I'll add a nuance here which I believe I've requested in the past...

    My OP discusses "your" benefiting from "my" decisions. To be selfish and save some time, I'd like to at least be able to benefit from my own decisions. Why do I need to approve the same eval statement in the same plugin across a number of sites? It would be great if Defender would save our decisions about ignoring or deleting unrecognized files, and ignoring warnings about specific plugin issues (Foo/2.5.1/1030/4). I say that plugin is OK for one of my sites, I can at least trust myself that it's OK for all of my sites - so Defender shouldn't bother me to make the exact same decision for every one of the sites.

    This is essentially a next step in the evolution of Automate. We have requests here about installing a specific plugin in all sites, and about configuring a single pluging similarly in all sites. Now I'm suggesting we just take another setting, decisions made in Defender, and propagating these to our other sites. The ability for others to benefit would be a huge next-step bonus.

    Thanks again.

  • splaquet
    • The Incredible Code Injector

    Funny you mention this, because I was pretty much thinking the same thing. Although, I'll have to tip my hat to your write up. Partly because I haven't had the time, but mostly due to your detailed explanation. I most likely would never have been able to articulate that as clearly as you, even if I had the time :stuck_out_tongue:

    Defender has recently been flagging a few plugins on my sites. I honestly don't have the skills to tell you if they're compromised, but they're paid... so I feel as though I should safely ignore them.

    If I ignore them, am I ignoring the file or the code snippet that was identified? As in, if I ignore the warning now, will a truly compromised file in the future go ignored?

    Currently these plugins are raising flags:
    - WPAllExport/Import
    - WooCommerce Authorize.net Processor

  • Nithin
    • Support Wizard

    Hi Tony G,

    In general highlighting, the eval functions alert in the File Scan is to bring the users attention and let them know there could be a function that could be exploited. It's up to the user to check, and ensure the code is safe, or contact support for any further assistance regarding cleaning up. If it's a trusted developer, then yes it would safe to disregard such notices.

    However, I do understand you, and a centralized metrics is really a good suggestion, however, I do see that factors to consider, and look upon are more in such use case, when compared to running a scan individually for each site. I'll make sure to bring your suggestion, and feedbacks into our team's attention so that we could discuss further regarding this, and see what further improvements could be looked upon.

    Regards,
    Nithin

  • Ivan
    • Developer

    Hi Tony G

    I think the part where user's decisions get synced across all his/her sites is the best way to go. Managing trust levels for users and weighing which amount of ignores are enough to permanently mark some line as false positive is something that is not very maintainable. There are situations where one user will ignore reported eval and other will not and will uninstall plugin even if it's false positive. We don't have a way to know or manage user's level of security related standards. Some users would eventually bash us cause Defender didn't warn them about xyz plugin having eval there and there.

    Kind regards,
    Ivan

    • splaquet
      • The Incredible Code Injector

      But Ivan,

      As correct as your statements are... there are MANY virus companies who maintain internal records, and allow for potential "false positive submissions".

      I realize that even the most accurate of coding can be compromised and turned malicious... but a threat report for many inexperienced folks can be truly concerning.

      Even I found one of my recent reports very concerning, because it flagged about a half dozen files from within our Auth.net CC processing plugin. I'm confident that the package is safe and accurate, but that didn't make it any less concerning when I saw the Defender report.

      • Ivan
        • Developer

        I'm just thinking from a development point of view, so my statements might come off as a bit technical.

        There is potential in what Tony G has described, it's just that it is a huge undertaking that would involve introducing a new role in a company and so on. Basically, it would turn us into one of those security oriented companies (which might not be a bad thing :wink: ).

        As James said below, we are working on global actions that might be a first step towards making process around Defender much more streamlined. (And not just Defender)

        We'll definitely take into consideration how we could improve Defender reports even more so that it doesn't make users freak out around stuff that is completely safe. I had the same experience with the security reports from various analyzers, it always worries me even if I know everything is safe.

        • Tony G
          • Mr. LetsFixTheWorld

          Ivan I really appreciate your note here because you are considering the request at a deep and realistic level.

          I've noted before that WPMU DEV took on a role as a security provider when it published Defender. The very name and concepts of protection and defense scream the intent and positioning of this product. You guys have accepted requests to include functionality available in other security plugins. You're already "one of those security oriented companies" until James says otherwise.

          If the company needs to draw lines between where it does and does not intend to serve as a security provider, I'm really hoping to see clear documentation about the current intents. Of course all such statements of positioning are subject to change. But it would help us all greatly if we know where Defender does and will leave off and where other solutions should begin. I won't ask Dev for a feature that I know is out of scope. I frequently try to encourage others to recognize the scope of these fine products and services, and to do the same.

          it's just that it is a huge undertaking that would involve introducing a new role in a company and so on

          Yeah, very true. Someone will need to be charged with processing reports, with some filter applied, so that some recommendation can be made. But we need something between "here" and "there". Where we are now, "here", we have a huge number of WPMU DEV clients spending their own time in multiple sites to evaluate potential threats, really without the proper skills to do so. And we all need to do this. Try to calculate the total wasted human hours. On the "there" side, we have someone helping to evaluate potential threats, UN-flagging some of the warnings, and saving all of the rest of us from dedicating attention to non-issues. There is value in "there". Might some WPMU DEV clients/members feel that an extra expense of $X per month is justified compared to the time and anguish, guessing about whether warnings are actual threats?

          Consider this method of helping us to get from here to there :

          - First, centralize all plugin warnings. That should be easy. If nothing else it will help all of us to get some metrics on how often specific plugins are flagged. We can expect that the same plugins will be flagged in the same files and near the same lines in each new version. Significant deviation (like a different line in the same version!) should create a true red flag for everyone.

          - Second, send an automated email to plugin authors to ask them about the line in question. That should be easy too using metadata from wp.org and from the plugins. If you don't get a response, leave the plugin flagged. A responsible developer doesn't want his/her plugin being flagged as a security issue. Yes, someone needs to process those responses, link them to reports, and then ensure that Defender no longer flags those specific plugin/version/file/line reports.

          - Third, automate the process where plugin authors can respond to issues. Use a Forminator form. Email a link to authors and let them come back to this site to just check a box : "Is the eval() on line 394 yours?" Answering yes can set a flag that will prevent Defender from warning all of us. (Yes, there are holes there too but again, we can plug the holes rather than avoiding the entire discussion.)

          That process eliminates all manual effort from WPMU DEV after its implementation. It gives us all a much better product. It draws significant attention to WPMU DEV for many reasons. It profiles the utility Forminator and Defender.

          We get all of these benefits when we collaborate and extend awareness of the company in the WordPress ecosystem.

          So someone is itching about sending emails, creating forms, and anything else above? OK, here's another way to implement that...

          Ensure there is a filter hook in Defender (probably is!) where we add a plugin that will do everything that I just suggested. Another plugin can handle the hook, centralize the reports, contact plugin authors, and then report back to Defender if there is reason to believe that a report is benign. Let someone else do the work and get the credit (and possibly the payment) for this service. Anyone at Dev suddenly feeling like they're missing an opportunity? :wink:

          OR, create an API end-point at Dev so that we can use our own tools to get metrics reported by Defender. This won't require each site to load a plugin as above. If, for example, I can query Dev through a web service and see that Defender just flagged 20 sites as having the same issues with the same version of a plugin, I can display that info on another site and WP members (and anyone else) just need to look at that site to see the metrics. Sure, again, there are holes here. But at least we're talking about possible solutions to get us from "here" to somewhere closer to "there".

          If none of that seems helpful (and I'd have a hard time believing none of that is), feel free to come up with other creative options to solve the problem. But there IS a problem. Let's put it on the table and see what we, collectively, can do about it.

          Thanks.

          • Ivan
            • Developer

            Tony G good stuff there. That looks like a very interesting development path for Defender to me personally. I really appreciate that you took the time to think through all this in such detail.

            We'll definitely discuss this internally and explore how that would fit our long term plan for Defender. You know I can't make any promises about realization though (at least not yet) :wink:

            Kind regards

  • christopher_wilson
    • Aithene

    Here's another user scenario to keep in mind.

    I'm not a developer, but I keep several websites up and running. As a full and/or part-time freelance Illustrator, Designer, and for the last decade, UX Designer, I've been doing this sort of stuff for nearly 20 years. I'm dependent on those with code skills to provide quality software that I can trust, and I'm willing and able to pay for it.

    I've also learned over the years through a couple of hacked websites and sites that go down due to no longer supported themes and plugins to keep my software up to date and as secure as I know how.

    So, when security software that I've been running for over a year out of the blue starts tagging code in a plugin I've been running for over two years, you can imagine that I get INCREDIBLY concerned. This is exactly what happened in the last Defender update. Without any warning of what to expect, it simply looked as though I'd been hacked. Someone had someone slipped into my plugins and started inserting malicious code. It took me several hours and a post to the friendly folks in this forum to decipher exactly what was happening, allay my fears, and finally allow me to get to bed hours past my bedtime for work the next day.

    Maybe on the next rollout, new checks can come with a warning stating that this is a new check, and that nothing has necessarily changed on your website.

  • jballen
    • Hack till it works

    It is very difficult to address the security scenarios for everyone as it really comes down to how much importance you place on security. Plugins are notoriously unsafe due to the reliance on the third party to maintain it, not to mention that many plugins do not properly erase all traces (database specifically) when deleted which of course opens even more attack vectors.

    More importantly, since a single instance of a plugin on just one site can be hacked, I don't see the feasibility of having a broad consensus that the flagged component is safe and can be ignored. Again, this really depends on your commitment to security and very likely the majority of those instances are benign. However, I much rather be warned rather than not and if you know your site well then it can be really quick to identify the level of the warning. Granted the sticker shot of the red flags is never comfortable to see, but I also thought it was made clear that they are scanning files that should not be located in the WP root directory (besides the php vulnerabilities).

    I am also in a very paranoid state right now as I am watching 26 banned IP login attempts that have occurred in the past 30 minutes. I think this is one of the most valuable aspects of Defender as now I am banning those IPs and collecting data that I hope will allow me to identify some IP ranges to block to prevent the attacks from escalating.

    • Tony G
      • Mr. LetsFixTheWorld

      In the process of telling us why you don't want to benefit from collective data, you verified your own specific need for it.

      I'd much rather have defender block all of those IPs for a period of time, knowing that some number of sites have already been maliciously probed by them, than to have to evaluate them all myself.

      With every initiative like this we need to take care not to do to much or too little. We trust Hustle not to annoy visitors. We trust Smush not to kill image resolution. We trust the Hub to update sites. Where these tools need refinement we ask for changes. That's the process. But I'd much rather have a process in place to refine than to have nothing at all, which is where we are now.

  • Tony G
    • Mr. LetsFixTheWorld

    Follow-up on this thread :

    I have a site that was in development at least a year ago, and I decided to clean it up, if for no reason other than to get the software current, to avoid hacks through vectors that might have been fixed long ago. After updating Defender and running a scan, there were 24 files flagged for containing eval or unserialize functions.

    I clicked the Select All checkbox and then Bulk > Ignore.

    Why? Because I'm not qualified to check the detailed usage of every plugin and their valid usage of these functions. What is the purpose of that warning? Am I supposed to post a note at WP.org to ask each author if they really put those functions in their code? Am I supposed to ask developer not to use these common and useful functions? Am I supposed to look through the support tickets to see if anyone else asked about this? What is Defender telling me to do? Be careful?

    A warning that is not actionable serves no purpose. I know my request for centralizing warnings would require extensive effort. My alternative for now is to take some action to stop Defender from doing those specific checks. If there isn't a filter hook to allow for skipping those tests, I'd be happy to make the change and submit a PR, but I suspect the change is trivial.

    Can we at least get THAT change in the product soon, so that after updates we don't see a bunch of inactionable warnings in the Hub, or Reports, or emails, or the Admin pages? All of these require manual attention to clear. That's a waste of our time. It's the opposite effect of one of WPMU DEVs core purposes, which is to save us time.

    Look, we all know that eval and other functions can be dangerous. But this "solution" to that problem was not well thought through. "Hey, let's warn people that they have an eval function in a file!" No, that's not the right solution for all of the reasons noted in this thread. Please allow us to disable what seems to me to be a bad, knee jerk resolution to a deeper problem. Then go back and think it through and come up with something else. But please don't make us live with a bad decision as though it was a good one that just needs refinement.

    OR, please explain how this really was a good decision and how the extra effort is justified on the part of every admin here, for every site we own. I'll accept some good reasoning there along with some agreement from our fellow member/colleagues. I fully accept that my decision to check All warnings and Ignore was wrong and irresponsible, but given the circumstances what else are we expected to do?

    Thanks.

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.