Sudden & Random Failure of IPN Forwarder for Pro Sites

Hello. I'm hoping @Aaron may have a minute to chime in here since it pertains to the IPN Forwarder script he wrote, and we are now seeing random IPN failure notices all of a sudden.

PayPal said in a recent email:

Please check your server that handles PayPal Instant Payment Notifications (IPN). IPNs sent to the following URL(s) are failing:

http://agreda.com/ipn/ipn-forwarder.php

Looking at my IPN history and the IPN forwarder logs, many notices are being sent successfully. But three times in the past few weeks we did have Pro Sites payments go through, only to have Pro/Supporter status withdrawan and the user notified of such.

For the successful IPNs the server response is always 200.

For all failed notifications, the server response is 503.

Here is just one recent sample of the numerous failed notifications in our IPN Forwarder logs:

[2014-02-09 15:31:59] Response Error: Unexpected transaction response while verifying with PayPal: 8cmd=_notify-validate&transaction_subject=Tripawds+Supporter+II%3A+%2449.00+USD+every+12+months&payment_date=08%3A46%3A18+Feb+08%2C+2014+PST&txn_type=express_checkout&last_name=zarembka&residence_country=US&item_name=Tripawds+Supporter+II%3A+%2449.00+USD+every+12+months&payment_gross=49.00&mc_currency=USD&payment_type=instant&protection_eligibility=Ineligible&verify_sign=AfGaKsU-pe.FM1ZMunPCl8f.W4k-AP705z5bVww.Nmivqx9ioYDp5Wiy&payer_status=verified&tax=0.00&payer_email=addressremoved%40email.com&txn_id=3J209966P89586542&quantity=1&receiver_email=myemailaddress%40mydomain.com&first_name=lori&payer_id=K8ST5P4DLD63S&receiver_id=YYC8CW7FKU5SY&contact_phone=123-456-7890&item_number=&handling_amount=0.00&payment_status=Completed&payment_fee=1.38&mc_fee=1.38&shipping=0.00&mc_gross=49.00&custom=site3_2744_2_12_49_USD_1391877976&charset=windows-1252&notify_version=3.7&ipn_track_id=bfc2546e1dc19

The majority of these failure notices begin with "8cmd=_notify-validate..." but there are also afew that begin "7cmd=_notify-validate..."

I have also noticed successful IPNs like the following in my PayPal IPN History, that are not in the IPN Forwarder logs:

transaction_subject=Tripawds Supporter&payment_date=02:12:37 Feb 08, 2014 PST&txn_type=subscr_payment&subscr_id=S-11422342E8831734D&last_name=Holliman&residence_country=US&item_name=Tripawds Supporter&payment_gross=25.00&mc_currency=USD&business=jcnjr@agreda.com&payment_type=instant&protection_eligibility=Ineligible&verify_sign=AEfmyQXvSSWIM09BDW02WJIykE-PAOe8KuVRXNa0inc8u5lhc..T4CUC&payer_status=verified&payer_email=emailaddress@removed.com&txn_id=7AT19828D4074983D&receiver_email=myemail@mydomain.com&first_name=Charles&payer_id=HWNETL39N4UGU&receiver_id=YYC8CW7FKU5SY&item_number=12&payment_status=Completed&payment_fee=0.85&mc_fee=0.85&mc_gross=25.00&custom=680_12_25.00_USD_1265674278&charset=windows-1252&notify_version=3.7&ipn_track_id=3fab499b4e089

Clearly something is amiss. We have added no new plugins recently. I have made no recent changes to Pro Sites settings, the IPN Forwarder array, nor the wp-config file. IPNs are still enabled in the PayPal account, with the forwarder script as the URL.

Any direction for further troubleshooting is greatly appreciated.

  • jcnjr

    PS: I apologize for the length of these posts, and only want to provide as much detail as possible...

    I have been wondering if this recent notice from PayPal will be having any affect on Pro Sites or the IPN Forwarder script that may necessitate edits to the plugin settings and/or forwarder script code.

    PayPal Says:

    ...Failure to make necessary changes to your PayPal integration may result in an inability to make API calls and/or receive payments through PayPal...

    Please make sure you are ready for this event by consulting with your technology team or individual(s) responsible for your PayPal integration...

    Because of a system upgrade, the following API endpoints are being updated:
    api.paypal.com
    api-3t.paypal.com
    svcs.paypal.com

    ...Merchants integrated in the following non-standard ways with PayPal’s API calls will be impacted when we update the API endpoints:
    Merchants calling our APIs with a hardcoded PayPal API endpoint IP address rather than using DNS resolution. See section A below.
    Merchants using HTTP methods other than GET, POST, DELETE and PUT. See section B below.
    Merchants using the HTTP 1.0 protocol. See section C below.
    Merchants whose firewall is configured to allow incoming and/or outgoing traffic from only a specific set of IP addresses will need to reconfigure their integration. See section D below...

  • jcnjr

    have you tried using the solutions provided on your following thread?

    Yes, I have made great effort to review various support threads reporting a similar issue, including the one of mine you referenced. The issue we are suddenly experiencing appears unrelated to the solutions offered elsewhere.

    I presume the 503 response on failed IPNs has something to do with it, but even after manually resending a notification from the PayPal dashboard, I will get a failure when I know the script is listening and functioning properly for other transactions.

    The only thing currently different on our server setup is that our server manager has enabled verbose logging to troubleshoot a brief nightly SQL outage which only occurs at a specific (late night) time and lasts only a couple minutes during backups. I wonder if this logging might have such an affect.

    I am also interested in Aaron's feedback regarding the API changes that PayPal says need to be made by March.

    Thank you.

  • jcnjr

    FYI: I'm only providing this update in an effort to provide as much detail as possible to assist with troubleshooting....

    Latest IPN Failure Notification:
    Latest delivery attempt date/time:
    HTTP Response: 503

    payment_cycle=Monthly&txn_type=recurring_payment_skipped&last_name=MacKenzie&next_payment_date=02:00:00 Feb 15, 2014 PST&residence_country=US&initial_payment_amount=0.00&rp_invoice_id=site3_2901_2_1_7_USD_1381368363&currency_code=USD&time_created=18:26:03 Oct 09, 2013 PDT&verify_sign=AzuSezQDT-PDgHlnPlulYA6vnlc4AxuSXgRDgdg3KaAOsts6vN5DA60B&period_type= Regular&payer_status=unverified&tax=0.00&payer_email=email@address.net&first_name=Jo-Anne&receiver_email=myemail@mydomain.com&payer_id=4KF84B32RG52E&product_type=1&shipping=0.00&amount_per_cycle=7.00&profile_status=Active&charset=windows-1252&notify_version=3.7&amount=7.00&outstanding_balance=0.00&recurring_payment_id=I-2PW7CPK2VFNE&product_name=Tripawds Supporter II: $7.00 USD each month&ipn_track_id=83d404195b274

    Recent Successful IPN:
    Latest delivery attempt date/time: 2/9/2014 17:59 PST
    HTTP Response: 200

    mc_gross=97.00&protection_eligibility=Eligible&address_status=confirmed&item_number1=TG-FBLOC1-0513&tax=0.00&payer_id=FJ3EL2NQ6NT2G&address_street=8533 Tunney Ave&payment_date=12:33:30 Feb 09, 2014 PST&option_selection1_1=12" x 30"&payment_status=Completed&charset=windows-1252&address_zip=91324&mc_shipping=0.00&mc_handling=0.00&first_name=KAROLE&mc_fee=2.43&address_country_code=US&address_name=Karole Velzy&notify_version=3.7&custom=&payer_status=verified&business=jerry@tripawds.com&address_country=United States&num_cart_items=1&mc_handling1=0.00&address_city=Northridge&verify_sign=AipzpiML256w9-akHONKruBmK6ovA0fFTyNkjTx5EKJqzXUJzHARjnaE&payer_email=customer@email.net&mc_shipping1=0.00&tax1=0.00&btn_id1=61397742&option_name1_1=Size&contact_phone=&txn_id=6YF77246VC815450G&payment_type=instant&last_name=VELZY&address_state=CA&item_name1=Farabloc EMF Shield Blanket&receiver_email=myemail@address.com&payment_fee=2.43&quantity1=1&receiver_id=YYC8CW7FKU5SY&txn_type=cart&mc_gross_1=97.00&mc_currency=USD&residence_country=US&transaction_subject=&payment_gross=97.00&ipn_track_id=4037cda0134fe

    Note how the successful IPN is for a product purchase from our store, not a Pro Sites payment or subscription creation. Reviewing our logs reveals that all purchase IPNs are successful yet the vast majority of Pro Sites related IPS seem to be failing all of a sudden.

    We're continuing to get IPN failures for Pro Sites and are having to deal with an increasing number of inquires from our paid "Supporter" bloggers so I'm desperately hoping someone can provide some feedback or suggestions.

    THanK YOU!

  • jcnjr

    What's wrong with this picture:
    2014-02-11 2:40:42 pm Expired email sent to subscriber@useremail.net
    2014-02-11 2:40:41 pm Pro Site status has been withdrawn.
    2014-01-11 1:30:40 pm Payment receipt email sent to subscriber@useremail.net
    2014-01-11 1:30:39 pm Pro Site status expiration extended until February 11, 2014.

    I'll tell you... on 02-10 the user's payment was successfully received but PayPal received a 503 when sending the IPN.

    I have confirmed that numerous IPNs are being successfully sent for single payments and non Pro Sites related recurring payments.

    Only Pro Sites subscriptions are getting the IPN failure. The IPN forwarder script is the only notification URL so this would indicate that something within the Supporter gateway is causing the failure since all other IPNs are successful.

    I'm desperately hoping someone can help provide some direction for further troubleshooting as we are manually managing these failures and inquiries by users daily now.

    In the mean time, here's my question/concern: What affect will manually extending the site's status considering subscriptions are still active? Will that matter once this issue is resolved and the IPNs start working working properly again?

    Thank you very much in advance for any help!

  • jcnjr

    Yes!

    2014-02-14 12:39:15 pm Payment receipt email sent...
    2014-02-14 12:39:14 pm Pro Site status expiration extended until March 14, 2014.
    2014-02-14 12:39:13 pm PayPal IPN "Completed" received: $5.00 recurring_payment

    Just had to share... the new IPN Forwarder script has everything playing nice with PayPal again.

    Many thanks once again to @Aaron and the WPMU Dev team for all your great work and support.