How to determine cause for Fundraising IPN faliure?

I read various topics discussing failed IPNs for the Fundraising plugin, but found no answer...

We received a donation via our Fundraising project yesterday, and got the notification from Paypal, but the gift has not posted in the plugin admin, nor has our progress bar updated.

Upon further investigation, I notice the IPN has failed to reach our site. Paypal reports the following:

Notification URL: http://tripawds.org/wdf-payment-return/paypal/
HTTP response code: 403
Delivery status: Retrying
No. of retries: 14

We received a donation last week and everything worked as it should. To my knowledge, nothing has changed on the server that might cause the Forbidden error for the IPN. We're running current versions of WordPress and the Fundraising plugin, and no new plugins, themes, or admin/settings changes have been added recently.

Since the notification URL is apparently not an actual file on which I can check permissions, how might I determine the cause for this failure? When I load the URL directly, a blank browser window displays with no errors and the Firefox console reports a "200 OK" HTTP response.

Thanks in advance for any help!

    Patrick

    Hi there @jcnjr

    I hope you had a great weekend!

    I presume that the .org domain has been mapped for a while; not just recently.

    I'm really not sure what could cause the IPN to suddenly fail if absolutely nothing has changed on your site or server. And I am not aware of any changes to PayPal APIs & such in recent days that could affect it.

    So I've asked our 2nd-level wiz-kids to pop in on this one to try to help out.

    jcnjr

    Thanks for the quick reply @Patrick

    Yes, the .org domain has been mapped since long before activating Fundraising.

    FYI: We have Pro Sites running on the network so I got concerned that its IPNs may be failing too. But I checked that PayPal account and there have been two sent successfully to tripawds.com after the failed one for tripawds.org above that is still Retrying.

    Jose

    Hi there James,

    Glad to hear from you again.

    This is the same site were I was working some time ago, right?

    As per the error details, this is some security layer in your server or site rejecting the request from PayPal (the URL returns 200 OK when called from the browser).

    Do you have some security plugin installed in your site? If so, you should check if a recent update added some sort of blocking per IP, user-agent, or similar that could block requests from specific origins.

    Also, there is a big chance that your server cofiguration was modified. I've run into issues with Apache mod_security module before.
    I recommend you to reach your hosting support and ask them if there is any recent modification to security settings and ask them about possible causes for the 403 forbidden resonse for PayPal.
    Server access logs may shed some light to the issue as well.

    Hope this helps.

    Please keep me posted on the progress.

    Cheers,
    José

    jcnjr

    FYI: On a separate note, at first glance the only entry in our server error logs that may even be remotely related is..

    [Mon Jun 15 09:03:52 2015] [error] [client 162.72.106.226] File does not exist: /home/tripawds/public_html/wp-content/plugins/fundraising/css/images/ui-anim_basic_16x16.gif, referer: http://tripawds.org/2015/06/unite-to-fight/

    I'm sure this has nothing to do with the IPN failure, but I confirmed there is no such image in the plugin folder and presume that should be resolved in the next update...I don't even know why that post would be calling that image. :-\

    Jose

    Jim,

    But why would the IPNs continue to be received successfully for tripawds.com (on the same server) yet fail for tripawds.org?

    I've seen weird behaviour on security layer rules. For mod_security for instance, it will decide if a request should be blocked based on several variables like request method (combined GET/POST normally cause issues), referrer, user agent, request body, etc.
    So, it is possible to even see some IPN calls working fine and some others failing in the same site.

    Regarding the logs, I meant the server access logs rather than the error logs.
    Anyway, it is good to know that there are no outstanding issues reported there.

    I'll wait for the feedback from your hosting. I assume that the option of a security plugin on your site can be discarded, right?

    Cheers,
    José

    jcnjr

    @Jose Said:

    Regarding the logs, I meant the server access logs

    Thanks for the clarification. The access log for the past 24 hours does include numerous references to the IPN notification URL, but they all have a 200 (OK) response. I'm attaching a text file for you to review please, to see if you notice anything odd. I've included only the relative entries from the log.

    What I don't understand is all the URLs with the plural ".../wdf-payment-return/paypals/" in what looks to be the referrer... ???

    Thanks again!

    jcnjr

    I would also like to to confirm the correct setting for the Notification URL in our PayPal account, since I just noticed the following discrepancy...

    The IPN "Retrying" message indicates:
    Notification URL http://tripawds.org/wdf-payment-return/paypal/

    But the URL entered in our Paypal account is:
    Notification URL http://tripawds.org/wp-admin/admin-ajax.php?action=wdf-ipn-return-paypal

    I'll presume this is correct, as I have never changed it since installing and setting up the Fundraising plugin. But has something changed? Should I leave it as is?

      Jose

      Jim,

      What I don't understand is all the URLs with the plural ".../wdf-payment-return/paypals/" in what looks to be the referrer... ???

      All those entries are just some test that I run directly from the browser to be sure that the IPN handler was responding as expected.
      I would like to see the access log for the date were the IPN call failed.

      The correct Notification URL is http://tripawds.org/wdf-payment-return/paypal/
      This was one of the modifications that I made to the plugin when you had issues some time ago.
      The former URL will still work but is prone to error. Nothing urgent, but you should change it if you have the chance.

      Any news from the hosting?
      Have you tried a new donation? does it fail consistently or just in some occasions?

      Cheers,
      José

    jcnjr

    Thanks again @Jose

    I've changed the notification URL as suggested to see if that helps. I guess Paypal won't let me resend the IPN until it's done retrying though.

    Unfortunately the access log for 06/14, the first day of failed IPNs, has been archived into the monthly log, which is a 1.5GB text file I'm unable to open. I'll try to capture the failure in upcoming daily logs.

    I will also try a new test donation if we don't receive one soon, to see if the failure is consistent.

    My last question above was specifically about the URLs in the access log that include paypals/ instead of paypal/ like this:

    190.224.47.4 - - [15/Jun/2015:17:00:29 -0500] "GET /files/2015/06/embrace-80x80.png HTTP/1.1" 200 7181 "http://tripawds.org/wdf-payment-return/paypals/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"

    Finally, I'm working with my server manager to determine if the PayPal IP address is getting blocked. Paypal has various IPs, however, so I need to l need to know which API endpoint the plugin is using...

    Jose

    Jim,

    My last question above was specifically about the URLs in the access log that include paypals/ instead of paypal/ ...

    It was me as well. I tested some variations of the url just in order to check if permalinks behaviour was correct.

    Finally, I'm working with my server manager to determine if the PayPal IP address is getting blocked.

    You should be able to get the IP originating the request in the access log. If you can't open it, please upload the zipped log somewhere and send me the link. I can look for the specific log for you

    Cheers,
    José

    jcnjr

    FYI: I just made a test donation and that IPN is also failing. Here are the access log entries, beginning with one for my IP address first, followed by PayPal...

    166.137.139.124 - - [16/Jun/2015:15:43:03 -0500] "GET /wdf-payment-return/paypal/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:38.0) Gecko/20100101 Firefox/38.0"

    173.0.81.1 - - [16/Jun/2015:16:11:51 -0500] "POST /wdf-payment-return/paypal/ HTTP/1.0" 403 3108 "-" "PayPal IPN ( https://www.paypal.com/ipn )"

    173.0.81.1 - - [16/Jun/2015:16:13:21 -0500] "POST /wdf-payment-return/paypal/ HTTP/1.0" 403 3108 "-" "PayPal IPN ( https://www.paypal.com/ipn )"

    21+ identical entries...

    The PayPal IP address is not being blocked by our server firewall config, but I have just added it to our whitelist anyway.

    Upon further research, I've learned that certain URLS ending with a slash/ may throw a 403 error if directory listing is denied. Could this be the cause? And if so, how would I allow listing since the notification URL is not an actual directory?

    Thanks again for the help!

    jcnjr

    @Jose said

    Thoughts?

    How familiar are you with Mod Security?

    Before I disable it for testing, I checked the mod_sec log and found the following. My server manager is unavailable right now, so I'd appreciate any feedback you can offer. Unfortunately, the Rule ID is listed as "unkown" ... :-\

    Domain | Source IP | Rule ID | Date Stamp
    tripawds.org | 173.0.81.1 | unknown | [16/Jun/2015:17:40:05 --0500]

    [16/Jun/2015:17:40:05 --0500] VYClxEWu9csAACbDSdgAAAAb 173.0.81.1 25715 69.174.245.203 80
    --db4b1c31-B--
    POST /wdf-payment-return/paypal/ HTTP/1.0
    Content-Type: application/x-www-form-urlencoded
    User-Agent: PayPal IPN ( https://www.paypal.com/ipn )
    Host: tripawds.org
    Content-Length: 934

    --db4b1c31-F--
    HTTP/1.1 403 Forbidden
    X-Powered-By: PHP/5.5.25
    Expires: Wed, 11 Jan 1984 05:00:00 GMT
    Cache-Control: no-cache, must-revalidate, max-age=0
    Pragma: no-cache
    Set-Cookie: PHPSESSID=59fbe4d43fd3dc4e9374cd5611a913d4; path=/
    Connection: close
    Content-Type: text/html; charset=utf-8

    --db4b1c31-H--
    Apache-Handler: fcgid-script
    Stopwatch: 1434494404729369 673789 (- - -)
    Stopwatch2: 1434494404729369 673789; combined=56, p1=42, p2=0, p3=0, p4=0, p5=14, sr=0, sw=0, l=0, gc=0
    Producer: ModSecurity for Apache/2.8.0 (http://www.modsecurity.org/).
    Server: Apache

    jcnjr

    @Jose Could you please consult with @Aaron & any other Domain Mapping lead developers to see if anything in the recent update of that plugin may in any way be related to the 403 response we are suddenly seeing for IPNs at tripawds.org?

    I have been doing extensive troubleshooting for this issue and note the following:

    1. Researching the IPN history indicates that the only change made on the server/site during the time frame when IPNs started failing was an update to certain WPMU Dev plugins including Domain Mapping.

    2. We have temporarily disabled all security (plugins and mod_security) on the server to test IPNs that continue to fail.

    3. IPNs are only failing on tripawds.org, a domain mapped to a subsite of the tripawds.com network.

    4. IPNs for Pro Sites on tripawds.com (with a different paypal account and IPN handler) continue to be received successfully.

    Thank you for your attention to this matter. Any suggestions for further troubleshooting are greatly appreciated!

    jcnjr

    UPDATE: After consulting with Paypal, my server manager and cPanel support, I need to implement PayPal's recommended fix...

    PayPal says:

    ...add User-Agent parameter in http header, such as below:
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: http://www.paypal.com','User-Agent: Tripawds Foundation'));
    or
    req.UserAgent="Tripawds Foundation".
    refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

    cPanel Support says:

    Both would be added in the connector script that reaches out to Paypal, and it would depend on what type of script you use as to which one to add. If you are using php you would use the first option, if you use .js you would put the second into the configuration files.

    @Jose Please provide direction for which Fundraising file I would need to edit, and how, to try this.

    Also looking forward to any confirmation that can be provided regarding Domain Mapping.

    Thanks again!

    Jose

    Hi there Jim,

    Sorry I didn't answer earlier. I was already off last Friday when you sent the message, and wasn't working during the weekend.

    Regarding white-listening PayPal's domains, I don't think that it would apply in this case. That's for other uses of the APIs.
    In this case, the referrer is always an IP number.

    I would like to have more context on PayPal's fix suggested above. It doesn't make sense for me. It seems something suitable for issues when calling to paypal, instead of issues when handling calls from. Still curious to read the full explanation.

    The Domain Mapping update can be a good clue.
    I'm pretty sure that the credentials that I have for your server are pretty out dated.
    I'll be glad to run some tests if you could please send me the following information via contact form :

    - In the subject field add "Attn: Jose”
    - Link back to this thread
    - Include WP admin/network access
    - Include FTP host and credentials.
    - Include any relevant URLS for your site

    On the contact form, select "I have a different question", this ensures it comes through and gets assigned to me.

    Cheers!
    José

    jcnjr

    Thank you @Jose

    After making some mod_security rule edits, the IPNs may be working again. (Paypal says it may take up to 16 transactions before we know for sure, since I had changed the IPN Notification URL in my account.)

    Please note that I had to revert to the old Notification URL for the IPNs to succeed! I hope this won't permanently change and will always work.

    This Notification URL Fails with 403:
    http://tripawds.org/wdf-payment-return/paypal/

    This Notification URL we are using Succeeds with 200 Response:
    http://tripawds.org/wp-admin/admin-ajax.php?action=wdf-ipn-return-paypal

    You can check both for yourself using the PayPal IPN Simulator if you like.

    Again, IPNs sent to Pro Sites at tripawds.com are working fine, it is only the mapped domain that had trouble receiving the IPNs.

    I'd like to keep this topic open until I know for sure that the issue has been resolved.

    PayPal was rather cryptic in their direction. Here is the entirety of the "fix" they recommended...

    Hi Jim,
    Thank you for contacting PayPal Merchant Technical Support.
    Yes, I see this IPN message with retrying status, http response is 403.
    For PayPal IP address list , please refer to https://ppmts.custhelp.com/app/answers/detail/a_id/92/
    Regarding user-agent, you can add User-Agent parameter in http header, such as below:
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: http://www.paypal.com','User-Agent: Tripawds Foundation'));
    or
    req.UserAgent="Tripawds Foundation".

    refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

    Refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html , http response 403 means "Forbidden". you can try to add IP address into firewall white list or add user-agent in header firstly.
    Hope this could be helpful, please feel free to contact us if any question.

    Sincerely,
    Samantha
    Merchant Technical Support
    PayPal, an eBay Company

    Obviously I never implemented this into any script. I did, however, add "Tripawds Foundation" to a mod_sec rule controlling User Agents. I also whitelisted the IP addresses for svcs.paypal.com... not sure which change fixed the IPN failure (if it is fixed) or if it wasn't something on Paypal's end all along...

    Jose

    Hey Jim,

    Regarding PayPal response, what can I say. In my years of experience, they rarely provided a helpful answer to my questions. By the look of it, I have the feeling that they simply don't read/understand the question. They just copy/paste some part of the documentation containing some of the words in question.

    I just tried the IPN Simulator and I got the same result. The new url is failing while the old url seems to work fine.

    If you send me the information requested above, I'll be glad to take a look and try to find the cause of the issue.

    Cheers,
    José

    jcnjr

    Thanks @Jose

    After requesting better confirmation from PayPal, I received the following suggestions

    Once you changed the IPN URL in your account, the change will take effect immediately. no delay. New IPN messages will sent to this new URL directly , but there are two exceptions:

    a. Your web developer add notify_url = http://tripawds.org/wdf-payment-return/paypal/ in website script. please let your web developer to search notify_url parameter in website script, then replace the link.

    This doesn't explain why the IPN Simulator continues to fail when sending messages to this notification URL. But perhaps it might explain why our IPNs started failing in the first place, since we did have the ajax URL set in our account in the time (and have since reverted to it).

    I may attempt to find and replace the notify_url as suggested, unless I hear back from you soon.

    b. Subscription payment. If IPN URL was http://tripawds.org/wdf-payment-return/paypal/ when you setup the subscription with buyer, then you changed the IPN URL , but following subscription payment will still sent to this old URL. You need to manually resent IPN message to your profile URL once it's failed after 4 days retrying.

    FYI: All our subscriptions were created under the original ajax URL, yet IPNs for the recurring donations are still failing even though we have reverted to that ajax URL after temporarily using the payment-return URL while troubleshooting.

    jcnjr

    I may attempt to find and replace the notify_url as suggested, unless I hear back from you soon.

    Well @Jose this turned out to be trickier than I expected. The new way the notification url is generated is rather confusing, to me at least.

    I'd like to try Paypal's suggestion to change the notify_url parameter, and did find this:
    paypal.php:161
    $nvp .= '&notify_url='.urlencode($this->ipn_url);

    But I doubt I should just change it there since I notice various references to that ipn_url...
    class.gateway.php:114
    $this->ipn_url = apply_filters('wdf_payment_return_url_' . $this->plugin_name, trailingslashit( home_url('wdf-payment-return/' . esc_attr($this->plugin_name))));

    This actually looks like the line that generates the new "wdf-payment-return" URL, so would it be correct to just change it here? Like this:
    class.gateway.php:114 hacked
    $this->ipn_url = apply_filters('wdf_payment_return_url_' . $this->plugin_name, trailingslashit( home_url('/wp-admin/admin-ajax.php?action=wdf-ipn-return-paypal' . esc_attr($this->plugin_name))));

    Or, is there some way, somewhere to just hard code the ajax notify_url somewhere since it is apparently working for us, to see if new IPNs no longer fail?

    Thanks!

    Jose

    Hey Jim,

    Sorry that I haven't had the chance to debug it yet. I was stuck with some critical bug fixing.

    First, you can hardcode the url in paypal.php:161. That should work. Other places where the url is used are jsut informative. For instance, in the instructions to setup the IPN in the admin dashboard.

    BUT, I must clarify that the IPN url was changed mainly because a lot of sites where having issues with mod security rules when using the ajax url. (there is a default rule that doesn't like the combination of GET and POST parameters when the request is not coming form a browser).

    So, you can try this change but at the end you may find another issues caused by the ajax url.
    It worth nothing to try anyway, just want to keep you informed.

    Please keep me posted on the results.
    As soon as I get the moment, I'll run some tests with the IPN simulator to try to spot the cause of the problem with the rewritten URL.

    Cheers,
    José

    jcnjr

    Thanks @Jose

    So, just like this...
    paypal.php:161
    $nvp .= '&notify_url='.urlencode(http://tripawds.org/wp-admin/admin-ajax.php?action=wdf-ipn-return-paypal);
    And ignore all the other instances of ipn_url?

    When was the new notify URL added? I believe we started seeing the failed IPNs since the last update.

    I am surprised we aren't having issues with the ajax URL, when others did. I am able to resend to it successfully, but new IPNs to the "return" URL fail with a 403... :-\

    With that said, this is the most recent "help" I received from PayPal...nothing i didn't know!

    IPN simulator can't send IPN message to http://tripawds.org/wdf-payment-return/paypal/ , the http response is 403 error.

    In this case, please check IPN related log in your server. there should be some error message or log relate to this 403 error, your developer can fix it on your web server according to the log info.
    Generally, if PayPal can't receive http response 200 from merchant's server, PayPal will continue to resend the message. We really want to help you to fix this issue, but we don't familiar with your web server configuration or environment, your developer need to work on it and fix it.

      Jose

      Jim,
      You were missing the quotes on the url string.
      This is the correct way:
      $nvp .= '&notify_url='.urlencode('http://tripawds.org/wp-admin/admin-ajax.php?action=wdf-ipn-return-paypal');

      When was the new notify URL added? I believe we started seeing the failed IPNs since the last update.

      The notify url parameter was always used. The new url was implemented like 6 months ago or so.

      The good thing about this error is that we are able to reproduce it using the IPN simulator. If you can use the ajax url as a workaround for now and give me a couple of days to look at it, that would be awesome

      Cheers,
      José

    jcnjr

    Thank you @Jose

    Looks like either of these will work:
    $nvp .= '&notify_url=http://tripawds.org/wp-admin/admin-ajax.php?action=wdf-ipn-return-paypal';
    or
    $nvp .= '&notify_url='.urlencode('http://tripawds.org/wp-admin/admin-ajax.php?action=wdf-ipn-return-paypal');

    I've edited my hack with your suggestion, and sent another IPN successfully immediately upon making a donation. Set up a daily recurring donation too, to ensure that's working.

    I plan to leave it this way until further notice, since it finally appears to be working again!

    Thanks again for all the help, and troubleshooting it further to see if there's a way I can avoid editing the plugin to work for us the next time it's upgraded.

    jcnjr

    The good thing about this error is that we are able to reproduce it using the IPN simulator.

    Just curious, you replicated the IPN failure using the Simulator with our Notification URL? Or, you were actually able to replicate the problem on another server too?

    give me a couple of days to look at it...

    No rush, our IPNs are working as expected for now, so I don't intend to change anything anytime soon.