Smart Crawl & Pinterest

I was just on support chat for a while with Majid Hamdani. At the end of it, he asked me to do a plugin test and I agreed. However, that can't be done in this case.

I can't do a plugin test on the live site (cuz it's live) nor can I do it on the dev site.

The live site is set up as a verified site for Pinterest. In order to do a plugin test on the dev site I would have to disconnect the live site from Pinterest and go through the whole process of registering the dev site on Pinterest for business and then verifying the dev site. Then, I would have to undo all of that and reverify the live site with Pinterest.

I considered using my personal website as a test, but that won't work either because I would have to register my personal site on Pinterest for business, which would be silly.

My conclusion, therefore, is that I cannot do a plugin test for this issue.

  • brentjohner

    Following up on my discussion with Majid ...

    Dreamhost says they are not blocking pinterestbot.

    Also, I have turned of every plugin that I dare to turn off on the live site and have not experienced any changes in behaviour on https://developers.pinterest.com/tools/url-debugger/

    However, I did note this statement on the debugger page: "If your page can't be validated, it means the required metadata wasn't detected on your page."

    Can somebody verify if that statement is true? I am not quite sure what I am looking for.

  • Lindeni Mahlalela

    Hello brentjohner

    I hope you are doing great today. Thank you for your patience with while we were looking into this issue. I have double checked everything and confirmed with the developers that SmartCrawl has basic support for all custom post types but doesn't have special support for products.

    According to the developers, SmartCrawl will not render the product OpenGraph data as it does not do anything specific products but it should render the og:title if any title is set in the title/meta for the products.

    So it seems the data expected by Pinterest for the rich pins will not be generated by SmartCrawl, fortunately I was able to come up with a quick solution that seem to solve the issue. I have created the following code snippet which you can use to generate the required OpenGraph meta for the products.

    add_action( 'wp_head', 'add_custom_opengraph_meta');
    function add_custom_opengraph_meta(){
        global $post;
        if ( $post->post_type == 'product' && function_exists( 'wc_get_product') ) {
            $product = wc_get_product( $post );
            $excerpt = get_the_excerpt();
            $excerpt = ( $excerpt != "" ) ? $excerpt : substr( $product->description, 0, 250); ?>
            <meta property="og:type" content="product" />
            <meta property="og:title" content="<?php echo $product->name; ?>" />
            <meta property="og:description" content="<?php echo esc_html( $excerpt ); ?>" />
            <meta property="og:url" content="<?php echo get_the_permalink( $post ); ?>"/>
            <meta property="og:site_name" content="<?php echo get_bloginfo( 'name' ); ?>" />
            <meta property="product:price:amount" content="<?php echo $product->get_price(); ?>" />
            <meta property="product:price:currency" content="<?php echo get_woocommerce_currency(); ?>" />
            <meta property="og:availability" content="<?php echo $product->stock_status; ?>" />
        <?php }
    }

    You can add this code in your current theme's functions.php file or create a new .php file and add the above code like so:

    <?php
    //above code goes here

    Then save the file and upload it to your website in the folder 'wp-content/mu-plugins' so that you have it like so:

    wp-content/mu-plugins/woo-opengraph-data.php

    Once you have this the code will start working without the need to activate it as it will be automatically loaded. Alternatively, you can simply download the attached 'woo-opengraph-data.zip' file and install it as a regular plugin by uploading it to your website via the plugin installer in "Plugins > Add New" page, after that you can activate the plugin, it should be named 'WooCommerce OpenGraph Data'. Here is how it looks on the plugins page:

    Once activated, this plugin does not add any menu or settings page that is why I recommend adding it as an MU Plugin, that is in the 'wp-content/mu-plugins' folder to keep your plugin list short.

    Once you have all this set, you can run the test again using the Rich Pins Validator and it should work. I have already tested it on my test website with simple products and it worked fine, I haven't tested with other product types and I hope this works for you as well, I tried this on your website and I got an error and according to the Pinterest Rich Pins FAQs, this should be the reason:

    If you're seeing We weren't able to retrieve any data from your URL, then Pinterest may be timing out before we can create a Pin on your site. Unfortunately, this isn't something we can fix on our end, and usually happens when a site is slow. This error also appears if your site blocks Amazon Web Services (AWS) requests. You'll have to remove this block before applying for Rich Pins.

    If you have any firewall blocking some requests then temporarily deactivate it and try again, this could be a plugin or some rewrite rules in .htaccess. Please let us know if you are having issues and we will be happy to assist in any way possible.

    Have a nice day.
    Lindeni

  • brentjohner

    Hi Lindeni.

    I truly appreciate the work you have done. However, I still can get this to work on either posts or products.

    The first failure this is.

    STEP ONE - CLAIM YOUR WEBSITE

    Pinterest's instructions are as follows:

    ---------------
    Pick claim option

    Option 1. Add HTML tag [I can't do this.]
    Paste this tag into the <head> section of your site’s index.html file

    <meta name="p:domain_verify" content="33963b921b8b9ba5c0f36054f56a167c"/>

    Option 2. Upload HTML file [I can do this.]
    Download this file and upload it to your website's root directory
    ---------------

    Of these two options, I have to choose option 1 because STEP 2 -VALIDATE YOUR PINS says the following.

    __________
    Choose any page on your site that you've added metadata to. Enter your chosen URL below and click Validate.
    Correct any problems you see with your metadata.
    Select one of three options depending on how you marked up your page:
    - HTML Tags, if you used Open Graph or Schema.org formats.*
    - oEmbed, if you used oEmbed format.
    - Shopify, if you have a Shopify site.
    Click Apply now. Your Rich Pins will show up within the hour.
    Note: You only need to validate and click apply for one link on your site to enable Rich Pins across your whole domain.

    If your page can't be validated, it means the required metadata wasn't detected on your page. See our Rich Pin documentation for more help.
    __________

    Please note the * which says: "HTML Tags, if you used Open Graph or Schema.org formats."

    This means that in STEP ONE I have to choose OPTION 1 (Add HTML tag)

    HERE IS WHERE I AM EXPERIENCING THE FIRST FAILURE ...

    ---------------
    See Pinterest's instructions here;
    https://help.pinterest.com/en/articles/claim-your-website
    Section: Use a Meta Tag
    ---------------

    In STEP ONE I am unable to claim my site using the meta tag instructions.

    Is it because of this sentence? "Add it to the <head> section of the index.html file of your website before the <body> section".

    https://racquetnetwork.com/index.html is 404

    I don't mean to be obtuse or difficult, there is just now way to move on to STEP 2 without completing STEP ONE.

    ---------------
    SUMMARY
    To be clear, I am able to CLAIM my site on Pinterest using the HTML file method. I am unable to claim it using the meta tag method.

    I believe I must use the meta tag method because Smart Crawl is using the Open Graph or Schema formats.

    So until this is resolved, I will be unable to claim my website in STEP ONE and therefore unable to move on to STEP 2 and STEP 3.

  • brentjohner

    STEP 2 - Validate Rich Pins
    https://developers.pinterest.com/tools/url-debugger/

    This resulted in failure. I tested the following URLs.

    A post:
    https://racquetnetwork.com/mens-squash-apparel/

    A product:
    https://racquetnetwork.com/mens-squash-apparel/

    According to the instructions:
    "If your page can't be validated, it means the required metadata wasn't detected on your page."

    I don't know how to check this. I used INSPECT but didn't see anything.

    Is somebody able to verify that the required metadata is in fact on the page?

    Again, sorry for being uninformed here.

  • brentjohner

    Hello.

    Re: Pinterest URL Validator
    https://developers.pinterest.com/tools/url-debugger/

    Pinterest says the following:
    "If your page can't be validated, it means the required metadata wasn't detected on your page."

    Can somebody please tell me whether or not the required metadata is showing on either of these pages?

    A post:
    https://racquetnetwork.com/mens-squash-apparel/

    A product:
    https://racquetnetwork.com/product/yonex-warmup-shirt-31018/

    I don't see it, but I may not be looking in the right place.

    Thank you.

  • Lindeni Mahlalela

    Hello brentjohner

    I hope you are doing great today. I am sorry for the inconveniences you are having here.

    With the plugin I have provided, the OpenGraph meta data is being generated for the products, it was already generating for posts but I have modified the plugin so that it will also generate meta data for the Articles, updated plugin attached.

    Here is how you can be sure that the meta data is generated on your pages. View the page source and then locate the following:

    a. Product

    <meta property="og:type" content="product" />
    <meta property="og:title" content="Technology Will Save Us Gamer DIY Kit" />
    <meta property="og:description" content="One of the permanent installations in the collection of Humble Masterpieces at the Museum of Modern Art in New York, this DIY gamer kit from London-based company Technology Will Save Us is equal parts gadget and design classic. " />
    <meta property="og:url" content="http://www.domain.com/urban/catalog/productdetail.jsp?id=37075900"/>
    <meta property="og:site_name" content="Urban Outfitters" />
    <meta property="product:price:amount" content="98.00" />
    <meta property="product:price:currency" content="USD" />
    <meta property="og:availability" content="instock" />

    Currently on your product pages:

    b. Article

    <meta property="og:type" content="article" />
    <meta property="og:title" content="Exploring Kyoto's Sagano Bamboo Forest - CNN.com" />
    <meta property="og:description" content="A constant inclusion on lists of "forests to see before you die," here's how to see the real thing." />
    <meta property="og:url" content="http://www.domain.com/2014/08/11/travel/sagano-bamboo-forest/" />
    <meta property="og:site_name" content="CNN.com" />
    <meta property="article:published_time" content="2014-08-12T00:01:56+00:00" />
    <meta property="article:author" content="CNN Karla Cripps" />

    Currently on your posts/articles:

    Even though the meta data is generated but the verification still fails for your pages and there could be various reasons for this, the most common one I found when searching are:

    1. Some required data may be missing, like for the product type, it is required that you have: title, description, amount, currency. I have checked on your website the products don't seem to have a price attached, this could be an issue on if some data is missing. I have tried setting the price for one product, to it didn't help, so it might be other issues.

    2. Pinterest Bot is not able to access your website due to a time out or Pinterest's bots are being blocked. Here is a quote from Pinterest detailing this:

    If you're seeing We weren't able to retrieve any data from your URL, then Pinterest may be timing out before we can create a Pin on your site. Unfortunately, this isn't something we can fix on our end, and usually happens when a site is slow. This error also appears if your site blocks Amazon Web Services (AWS) requests. You'll have to remove this block before applying for Rich Pins.

    This can be solved by disabling all security rules that block bots from accessing your site. This could be security plugins, .htaccess rules, Web Application Firewalls and or other rules set by your host. It is possible that there website is too slow and Pinterest keeps timing out when trying to check your website. In order to test if this is the case, you can simply copy the source code of the poduct page, paste it in a new HTML document an upload it to a website hosted on another server and try the validation again, if it passes then the issue is with your current server. I do believe that the website is too slow as it takes a long time to render any of the pages, this could be the problem.

    Please see the answer at the bottom of this page for more information about this error.

    3. It might also be possible that there were changes made in the website's settings after it was first verified by Pinterest. This could may be a change on the domain from www to non-www or from https to non https or the change of the Site name itself. You may fix this by following this guide: https://www.theprofessionalwomanblog.com/pinterest-rich-pin-error/

    At this point, the meta data is generated the problem lies within the verification, and so there is nothing much we can do from our side besides advice you to debug based on the above two points and also contact your host to find out if they are not block AWS services or bots. Also check if you have any security rules that may block Bots from accessing your site.

    Please also note that, the exact code works perfectly on my test website, so the issue is not the code but certain aspects of the server like speed and or blocked restricted access.

    I hope this helps.

    Have a nice day.
    Lindeni

  • brentjohner

    This is excellent feedback. I will work on this to see if I can solve the problem.

    One small issue ...

    I now have this error ...
    Warning: count(): Parameter must be an array or an object that implements Countable in /home/racquetnetwork/racquetnetwork.com/wp-includes/post-template.php on line 284

    ... appearing at the top of this page
    https://racquetnetwork.com/

    It appeared after you worked on the site today. I'm guessing it's related?

    Thanks in advance for taking a look.

  • brentjohner

    Here is the transcript from my chat with Dreamhost. It appears unlikely that this is a blocked bot issue.

    Chat started on 23 Mar 2018, 11:56 PM (GMT+0)
    (11:56:20) *** Racquet joined the chat ***
    (11:56:20) Racquet: The folks at WPMU DEV asked me to ask you a question ... "contact your host to find out if they are not block AWS services or bots. Also check if you have any security rules that may block Bots from accessing your site."
    (11:56:56) Racquet: We are trying to use the URL validator on this page: https://developers.pinterest.com/tools/url-debugger/
    (11:57:26) Racquet: We are tying to verify this URL: https://racquetnetwork.com/why-do-strings-break/
    (11:57:36) Racquet: It will not validate
    (11:58:16) Racquet: The two possible explanations are 1) the site is too slow and 2) the Pinterest bot is blocked
    (11:58:58) Racquet: I want to rule out at least one of these two options.
    (12:12:07) *** Heather D joined the chat ***
    (12:12:41) Heather D: Hi there! I can help you out with this. We do not block any bots specifically by default. If you can give us the IP of the bot in question we can check to see if that itself is blocked.
    (12:13:26) Racquet: I'm not sure how I would find that out. It is a Pinterest bot. Does that help?
    (12:14:43) Heather D: You would need to ask Pinterest in that case.
    (12:16:52) Racquet: OK. Their other question was about AWS services.
    (12:17:51) Heather D: Right, the same applies for that. We don't block it by default, but if you can let us know the IP that the request is coming from we can check for that individually.
    (12:18:52) Racquet: Ok. So I really can't answer their question because you can't answer my question because I don't have any idea how to get an answer to your question.
    (12:19:19) Racquet: Thank you. Have a good evening.
    (12:22:28) Heather D: You too!
    (12:22:29) *** Heather D left the chat ***
    (12:23:08) *** Racquet left the chat ***

  • Lindeni Mahlalela

    Hello brentjohner

    I hope you are doing great today. Thank you for the feedback and for contacting your host about this issue.

    Unfortunately, we also do not know what IP addresses are used by Pinterest but here is what I found and I am not sure how reliable this is: https://www.distilnetworks.com/bot-directory/bot/pinterest-bot/ there is also a list of User Agents here https://help.pinterest.com/en/articles/about-pinterest-crawler-0 which are sometimes blocked instead of the IP addresses.

    Warning: count(): Parameter must be an array or an object that implements Countable in /home/racquetnetwork/racquetnetwork.com/wp-includes/post-template.php on line 284

    I am not sure how this relates to the code I have added but I think it is because I have enabled debug mode in WordPress by adding these lines:

    define('WP_DEBUG', true);
    define('WP_DEBUG_LOG', true);

    Please look for these lines in wp-config.php and if they exist, try to make sure there is a line that looks like this:

    define("WP_DEBUG_DISPLAY", false);

    This line will make WordPress hide these warnings. If it still doesn't hide then remove these line or simply change:

    define('WP_DEBUG', true);

    To:

    define('WP_DEBUG', false);

    And then try again with the code I have provided. I have found on the internet that there is a known issue in PHP7.2 that causes this for, unfortunately this affects WordPress core as it affects you can see the reported file is in the WordPress Core includes folder. I also think you should not worry about this as it is only a Warning and should not stop your website from working, it may only be an issue if it is displayed on your front end in which case you have to disable debug or activate debug logging and hiding warnings by adding the defines as mentioned above.

    It will be a reason to worry if it were a fatal error, warnings are not of a serious problem and you should see a lot of them if some plugins and themes are not up to date as there will likely be a lot of deprecated functions especially with PHP7.2, many themes and plugins are still way behind, but you should not worry about warning just suppress them so they are written in debug file and hidden from the normal page output by adding the above defines in wp-config.php file.

    I hope this helps, please let us know if you have any further issues or concerns and we will be happy to assist in any way possible.

    Have a nice day.
    Lindeni

  • brentjohner

    I don't know. This issue is above my pay grade. I may have to drop it.

    I just tried to validate again using this tool:
    https://developers.pinterest.com/tools/url-debugger/

    This is the URL I submitted for a POST:
    https://racquetnetwork.com/ladies-pickleball-shoes-sizes-11-12-and-13/

    This is the error message I got:
    "The data we scraped from your site could not be validated. Please make sure all required tags are present and you aren't serving different pages depending on different user agents."

  • Lindeni Mahlalela

    Hello brentjohner

    I hope you are doing great today. I am so sorry for all the inconveniences caused.

    This is the message I got:
    "Your Pins have been validated! Apply to enable your Rich Pins."

    So we seem to be part way there. Rich pins appear to be working for products. I will do some testing now on Pintrest.

    I thought this was good news and I was wondering if you got your application through. I am not sure if you know that you can get the application approved on Pinterest with justt the one post, you do not have to have all your products setup with OpenGraph data to have the application approved.

    Did you submit the application and how did it go?

    I have replied on the other ticket regarding SmartCrawl. We have tried to help where ever possible and I have gone beyond the scope of this support forum and beyond my own duties to provide a solution I thought would help. I am really sorry if all that did not help. We will really appreciate your feedback with regards to SmartCrawl so please head on to our Features and Feedback section and submit your feedback and or request features you think will make SmartCrawl better for you and others as well.

    Please note that in this support forum we cannot provide full working solutions as that will require a lot of research and custom coding even if its small features, but we always try to provide solutions whenever possible. In the future for such custom work you may like to hire a professional developer via the Jobs Board instead of waiting for tech support which has limited support scope.

    Please let us know should you have any further concerns or feedback and we will be happy to assist.

    Have a nice day.
    Lindeni.