How to decode wp_remote_get content

Hey there,

I am pulling in a feed using the wp_remote_get() function.

However, how do I decode/make-usable the code I retrieve?

I have tried this before:

$data = file_get_contents('http:/www.example.com/feed-url');
        $feed_content_xml = new SimpleXmlElement( $data );

Which works great, however, it doesn't look like I can use the wp_remote_get() function with the SimpleXmlElement() function.

Sincerely,
Mika

  • Michael Bissett

    Hey @Mika, hope you're doing well today! :slight_smile:

    Could you elaborate as to what information you're looking to use from the feed? I'm curious as to the use of the wp_remote_get function in this instance, seeing as there's the fetch_feed function available:

    https://codex.wordpress.org/Function_Reference/fetch_feed

    Perhaps there's something I'm not aware of here? :slight_smile:

    Please advise,
    Michael

  • Imperative Ideas

    This is nearly descriptive enough to be useful.

    wp_remote_get() as you are likely aware returns a raw response from the server. Before we represent the XML feed as an object, let's have a look at what is being returned.

    <?php
    echo '<pre>';
    print_r($data);
    echo '</pre>';
    ?>

    What is being output? Once we see the raw data being returned by WordPress, then we can talk about how to parse it. On its own, wp_remote_get() isn't going to return just the XML data, it's also going to return the header information. You probably need something more like this:

    $feed_content_xml = new SimpleXmlElement( $data['body'] );
    • Mika

      @Michael and @Imperative Ideas, thanks a lot for your replys.

      The reason why I am referring to the wp_remote_get() function is because I read this article from Yoast: https://yoast.com/wordpress-http-api/.

      The way I understand it is, that the wp_remote_get() function is extremely flexible ( speaking of the many server setups ), so this might prevent issues from my plugin users along the way. I have used cURL so far, however, the wp_remote_get() function seems more flexible.

      Nevertheless, the fetch_feed() function also sounds very interesting. Does any body know ( @Jose, blink, blink ), which function is best to use to fetch feeds and use specific parts of the content ( such as the title, description, url, etc): cURL, wp_remote_get() or fetch_feed()?

      Here I am specifically looking for the function which is fastest and require less memory resource.

      Once we have a winner, I would also love to see an example of how to fetch the feed and then use the content along the lines what Imperative Ideas suggested.

      Sincerely,
      Mika

  • Jose

    Hi there Mika,

    There isn't a noticeable performance difference among the methods that you mention.

    Using cURL directly in your code is not a good approach because you need to take care of every single detail.

    Both, wp_remote_get() and fetch_feed() are using cURL in the background as the default transport method.

    wp_remote_get() will use cURL as the default transport and fallback to socket streams connections in case cURL is not enabled in the server. It will also wrap some basic sanitization, error handling, and some other stuff that is absolutely necessary.

    fetch_feed() relies on SimplePie library and it goes a step beyond. It takes care of parsing your XML feed.
    If you are willing to fetch standard RSS feeds, this is definitively a good option.
    Now, if you are fetching data with a custom format, you may want to use wp_remote_get() and add your custom parsing logic to the result.

    Hope this helps :slight_smile:

    Let me know if you have further questions.

    Cheers,
    Jose

    • Mika

      Hi @Jose

      Amazing, now are talking. Thank you so much for your description. It was very, very helpful!!

      In that case, I will go ahead and use the feed_fetch function.

      I have created a function that is hooked to the wp_add_dashboard_widget() ( That is where the feed will be shown for now ). At this point I am echoing everything, which seems to work fine in the dashboard widget.

      However, if I hook the the same function into a shortcode (add_shortcode), it doesn't work properly because, I have to return the values instead of echoing them when working with shortcodes.

      Is there a smart way I can make the functions work both as a shortcode and as a dashboard widget, or do I need to make a new function and use return instead of echo in the new code, and the hook it into the add_shortcode() function?

      I hope it make sense :slight_smile:

      Sincerely,
      Mika

      • Jose

        Yes, the best approach is to create a specific handler for each one.

        Something like this:

        function fetch_and_process_feed(){
            $result = //Do al your stuff to get the result.
            return $result;
        }
        
        function handle_shortcode_feed(){
            $output = fetch_and_process_feed();
            //Do specific things on for shortcode $output if needed.
            return $output;
        }
        
        function handle_dsahboard_widget_feed(){
            $output = fetch_and_process_feed();
            //Do specific things on for dashboard $output if needed.
            echo $output;
        }

        You then would hook each action with the specific handler :slight_smile:

        Cheers,
        Jose

        • Mika

          Hey @Jose

          I just wanted to hear, how can I add save following code to a variable ( as you referred to be your example function: fetch_and_process_feed() ).

          function fetch_recent_feed(){
                  // Get a SimplePie feed object from the specified feed source.
                  $rss = fetch_feed( 'http://myfeedurl.dk' );
          
                  $maxitems = 0;
          
                  if ( ! is_wp_error( $rss ) ) : // Checks that the object is created correctly
          
                      // Limit feeds to 5.
                      $maxitems = $rss->get_item_quantity( 10 ); 
          
                      // Build an array of all the items, starting with element 0 (first element).
                      $rss_items = $rss->get_items( 0, $maxitems );
          
                      // Sort order to show latest post
                      $rss->enable_order_by_date( false );
          
                  endif;
          }

          Sincerely,
          Mika

          • Jose

            Hey Mika,

            Not sure if I follow you.

            Do you mean how to store the feed items in the DB?
            If that's your question, you can serialize the data and save it as an option:

            $serialized_items = maybe_serialize($rss_items);
            update_option('feed_' . $feed_ID . '_items', $serialized_items);

            BUT, you should take into account that SimplePie provides a builtin cache feature. So you may not need to store anything but just control the cache lifetime by using the filter hook 'wp_feed_cache_transient_lifetime'.

            Let me know if I understood your question correctly :slight_smile:

            Cheers,
            Jose

          • Mika

            Hey Jose.

            Althroguh, I am super stocked learning about the serialize function ( like a whole lot ), that is not quite what I had in mind.

            I am referring to your email, where you explained me how I would go about making three functions. One function to get the feed, one function that shows the feed using "echo", and one function that shows the feed using return ( for shortcode purposes).

            However, my question is, how do I add the "fetch_recent_feed() to the other two functions? Do I just make it a global in the two other functions and then just call the function, or how would I go about it? :slight_smile:

            I am so sorry, if I do not make sense. If not, I will try to write it to you in Danish since you are mastering it so well :wink:

            Thanks Jose! I wish you a wonderful weekend!

            Sincerely,
            Mika

          • Mika

            Hi @Jose

            I did found a great way to accomplish what I wanted to do. I added

            ob_start();

            In the beginning of my function and

            $output = ob_get_clean();
            
                if( is_page() ) {
                    return $output;
                } else {
                    echo $output;
                }

            In the end, which made me able to create a shortcode and use the same function :slight_smile:

            Ps. do you know if is_page() is the best condition to use to check if it is on the page? And, how can I add a condition to see if the shortcode is active in sidebar?

            Sincerely,
            Mika

  • Michael Bissett

    Hey @Mika,

    Are you needing any more help with this then? Didn't see that this thread was marked as resolved, and just wanted to make sure of that. :slight_smile:

    To answer your other questions:

    do you know if is_page() is the best condition to use to check if it is on the page?

    is_page() is meant to check if a page is being displayed, it's not designed to check to see if a piece of content is on a page.

    And, how can I add a condition to see if the shortcode is active in sidebar?

    This article should help to lay out how this can be done:

    http://code.tutsplus.com/articles/quick-tip-improving-shortcodes-with-the-has_shortcode-function--wp-21007

    Kind Regards,
    Michael

  • Mika

    Hey @Michael

    Thanks for your reply. I think I should be good, however, I might have one last question.

    SimplePie uses a cache for 12 hours. Does that mean that my feed only is updated once every 12 hour? If not, could you please explain how the SimplePie works practically with WordPress.

    The code from TutsPlus is great, however, it only target the content in the main query, not the content in a sidebar widget. So this function doesn't really seems to be doing the job.

  • Jose

    Hey Mika,

    Was off for the weekend. I see that you already made some decisions, but I'll try to answer anyway :slight_smile:

    However, my question is, how do I add the "fetch_recent_feed() to the other two functions? Do I just make it a global in the two other functions and then just call the function, or how would I go about it? :slight_smile:

    If I understand correctly, your question is related to scopes. You definitively need to learn and get a clear understanding of this subject.

    In your case, I don't see the need of using the output buffer. You should stick to the original approach and simply do something like this:

    function fetch_recent_feed(){
            // Get a SimplePie feed object from the specified feed source.
            $rss = fetch_feed( 'http://myfeedurl.dk' );
    
            $maxitems = 0;
    
            if ( ! is_wp_error( $rss ) ) : // Checks that the object is created correctly
    
                // Limit feeds to 5.
                $maxitems = $rss->get_item_quantity( 10 ); 
    
                // Build an array of all the items, starting with element 0 (first element).
                $rss_items = $rss->get_items( 0, $maxitems );
    
                return $rss_items;// Return the array of items on success. 
    
            endif;
    
            return false;// Return false on error.
    }
    
    function fetch_and_process_feed(){
        $result = fetch_recent_feed();// You can call the method directly because both are on the same scope.
        $result = do_some_more_processing_if_needed();// Once you have the array, you may need to do further filtering, sorting, or something.
        return $result;
    }
    
    function handle_shortcode_feed(){
        $output = fetch_and_process_feed();
        //Do specific things on for shortcode $output if needed.
        return $output;
    }
    
    function handle_dsahboard_widget_feed(){
        $output = fetch_and_process_feed();
        //Do specific things on for dashboard $output if needed.
        echo $output;
    }

    do you know if is_page() is the best condition to use to check if it is on the page?

    Yes, this is meant to provide you a way to know if you are loading a Page (when using it without parameters) or if you are loading a specific page (when using page id, title or slug in the first parameter).

    And, how can I add a condition to see if the shortcode is active in sidebar?

    I don't see the point on this one. In our approach, handle_shortcode_feed() will be executed only when the shortcode is being rendered. So, we should assume that every time that the handler is called it is because the shortcode is active.

    SimplePie uses a cache for 12 hours. Does that mean that my feed only is updated once every 12 hour? If not, could you please explain how the SimplePie works practically with WordPress.

    Yes, this means that -by default- SimplePie will store the result for 12 hours to save resources. Avoiding the need to call the feed url each time.
    If you need the feed to be fresh or to reduce the duration of the cache, you simply need to use the following hook:

    function modify_simplepie_cache_duration( $seconds ) {
      // change cache duration to 1 hour. It is expressed in seconds.
      return 3600;
    }
    
    add_filter( 'wp_feed_cache_transient_lifetime' , 'modify_simplepie_cache_duration' );

    Hope this helps. Take care :slight_smile:

    Jose

    • Mika

      Hi Jose.
      Amazing, thanks a lot for your answer. I will look into the practical application of it!!

      It totally make sense!! :smiley:

      Regarding the last point with the filter. I talked with Vinod today about that. He suggested the same approach as you, however, I said to do it like this:

      // change the default feed cache recreation period to 1 minute
              add_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
              function rocapress_change_feed_cache_interval( $seconds ) {
                  return 60;
              }
      
              // Get a SimplePie feed object from the specified feed source.
              $feed = fetch_feed(  'http://myfeedurl.dk' );
      
              // remove the custom feed internal
              remove_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );

      Please notice the remove_filer function in the end. He said that this was added so that it did not cause unnecessary page load delay on other pages ( if I understood him correctly ). With your limited knowledge of why he suggested the remove_filter function, could you shed some light on why it is a good idea to use it? :slight_smile:

      Sincerely,
      Mika

      • Jose

        Hey Mika,
        Normally, you would want to remove hook listeners if it is no more necessary in the current execution. i.e: when you want to ensure that a filter is applied only once.

        In this particular case, you are not doing anything complicated nor resource consuming in your handler. You are always returning the same hard coded value, so there is no harm if the filter gets applied more than once.

        In any case, if you want to remove the handler, you are doing it wrong in the example above.
        You are adding the handler and removing it in the same routine, so the filter won't be applied at all.
        The remove_filter() method should be called AFTER the hook has being applied.
        Something like this would work:

        // change the default feed cache recreation period to 1 minute
                add_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
                function rocapress_change_feed_cache_interval( $seconds ) {
                    // remove the custom feed internal
                    remove_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
                    return 60;
                }
        
                // Get a SimplePie feed object from the specified feed source.
                $feed = fetch_feed(  'http://myfeedurl.dk' );

        Does it makes sense for you?

        • Mika

          Hi Jose.

          Yes, totally. I have just made the change, and I really appreciate your feedback!

          If I may ask, and I truly apologize for my extreme ignorance, could you please show me how to populate the function handle_shortcode_feed() ( from the topic a bit above ), so I can see how you would do it practically.

          I have tried it in several ways, but nothing seems to be working so I must be doing something wrong.

          Thank you so so much Jose! I am really learning much here, so I appreciate your time and patience!

          Sincerely,
          Mika

          • Jose

            Mika,
            Your questions are very welcome. No need to apologize :slight_smile:

            My example above is just a draft. Normally a real world implementation has more lines than that, because there is always something specific for each case that need to be handled.

            That said, the example should work as is. As long as the $output variable is populated correctly the shortcode should show something in the page.
            I recommend you to add a var_dump trace in order to see where is it failing:

            function handle_shortcode_feed(){
                $output = fetch_and_process_feed();
                echo '<pre>';// open pre tag just for pretty print.
                var_dump($output );
                echo '</pre>';// close pre tag just for pretty print.
                die(); // We stop the execution to ensure that anything else is interfering with our debugging result.
                //Do specific things on for shortcode $output if needed.
                return $output;
            }

            I'm assuming of course that you are including the add_shortcode() call somewhere. Something like this:
            add_shortcode( feed_shortcode', 'handle_shortcode_feed );u

            You can post all your code here if you want me to take a look. :slight_smile:

            Cheers,
            Jose

          • Mika

            Amazing, thanks Jose!

            I will brew a bit on the code and get back to you with it asap!

            By the way, I am considering writing extensions to the plugin in the future, I assume that your way ( with the several functions ) make this possible. Would I use the apply_filters function for each of the returned ( or echoed ) values? or where would I add the filters to make the plugin more extendable? :slight_smile:

            Sincerely,
            Mika :slight_smile:

          • Jose

            Hey Mika,

            Actually, while having functionality separated in different functions is a good practice and makes the code easier to read and maintain, it is not directly related with being extensible.

            You could add the apply_filters or do_action calls even if you have a huge monolithic piece of code.

            Made that clarification, only you will know where you want to provide the ability to hook/extend your code. It's doesn't need to be necessarily on a return.

            At a first glance, I see two good opportunities.
            First, to provide a way to alter the url. This would allow to send custom query args on the fly:

            // Get a SimplePie feed object from the specified feed source.
            $rss = fetch_feed( apply_filters('my_plugin_name_fetch_feed_url', 'http://myfeedurl.dk') );

            Second, to provide a way to filter the array of items returned by the feed:

            // Build an array of all the items, starting with element 0 (first element).
             $rss_items = apply_filters('my_plugin_name_rss_items', $rss->get_items( 0, $maxitems ));

            I'm sure you will find some other places where a hook would be handy. :slight_smile:

            Cheers,
            Jose

          • Mika

            Hi Jose.

            That makes completely sense!!!

            I will consider those changes as I continue the development of the plugin.

            This is the full code so far ( the functions names, comments, etc will be updated as this is a rough draft, which I managed to get to work. Please note that I had to add the ob_start() and ob_get_clean() to output the shortcode):

            /* Add a widget to the dashboard. */
            add_action( 'wp_dashboard_setup', 'rocapress_dashboard_widget' );
            function rocapress_dashboard_widget() {
                wp_add_dashboard_widget( 'rocapress_dashboard_widget', 'Seneste 10 example Indlæg', 'rocapress_dashboard_widget_func' );
            }
            
            function rp_fetch_feed() {
            
                        // change the default feed cache recreation period to 1 minute
                    add_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
                    function rocapress_change_feed_cache_interval( $seconds ) {
                        // remove the custom feed internal
                        remove_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
                        return 60;
                    }
            
                    // Get a SimplePie feed object from the specified feed source.
                    $feed = fetch_feed( 'http://www.example.dk/forums/aggregaterss.aspx?Mode=0' );
            
                    // Enable SimplePie's feed cache system
                    $feed->enable_cache( true );
            
                    if ( ! is_wp_error( $feed ) ) : // Checks that the object is created correctly
            
                        // Limit feeds to 5.
                        $maxitems = $feed->get_item_quantity( 10 ); 
            
                        // Build an array of all the items, starting with element 0 (first element).
                        $feed_items = $feed->get_items( 0, $maxitems );
            
                        // Sort order to show latest post
                        $feed->enable_order_by_date( false );
            
                        return $feed_items;
            
                    endif;
            
                        return false;
            
            }
            
            function rocapress_shortcode(){ 
            
                $feed_items = rp_fetch_feed();
            
                ob_start();
            
                echo '<ol>';// open pre tag just for pretty print.
            
                foreach ( $feed_items as $item ) : ?>
                                <li>
                                    <a href="<?php echo esc_url( $item->get_permalink() ); ?>"
                                        title="Klik for at se indlægget" target="_blank">
                                        <?php echo esc_html( $item->get_title() ); ?>
            
                                    </a>
                                </li>
                <?php endforeach; 
            
                echo '</ol>';// close pre tag just for pretty print.
            
                $output = ob_get_clean();
                return $output;
            }
            
            /* function that retrives the latest 10 feed from example.dk */
            function rocapress_dashboard_widget_func() {
            
                $feed_items = rp_fetch_feed();
            
                echo '<ol>';// open pre tag just for pretty print.
            
                foreach ( $feed_items as $item ) : ?>
                                <li>
                                    <a href="<?php echo esc_url( $item->get_permalink() ); ?>"
                                        title="Klik for at se indlægget" target="_blank">
                                        <?php echo esc_html( $item->get_title() ); ?>
            
                                    </a>
                                </li>
                <?php endforeach; 
            
                echo '</ol>';// close pre tag just for pretty print.
            }
            
            /* add shortcode so the user can display the feeds in posts, pages and the widget area */
            add_shortcode( 'example_latest_feed', 'rocapress_shortcode' );
            
            // Enable shortcodes in the widget text area so the [example_latest_feed] can be added and display properly.
            add_filter('widget_text', 'do_shortcode');
            
            ?>

            Ps. I did not add your example function fetch_and_process_feed() function yet, I will look into that after your feedback :smiley:

            I am looking forward to hear you feedback :slight_smile:

            Sincerely,
            Mika

          • Jose

            Hey Mika,

            Just tried your code in my own environment and it works :slight_smile:
            That's always a good signal!

            Everything looks pretty good to me.

            Regarding the use of the output buffer, it will depend on your particular needs.
            In this case, there is no noticeable difference between using ob_start() / ob_get_clean() vs assigning the output to a variable like this:

            function rocapress_shortcode(){
            
                $feed_items = rp_fetch_feed();
            
                $output = '<ol>';// open pre tag just for pretty print.
            
                foreach ( $feed_items as $item ){
                    $output .=  '<li><a href=""' . esc_url( $item->get_permalink() ) . '" title="Klik for at se indlægget" target="_blank">' . esc_html( $item->get_title() ) . '</a></li>';
                }
            
                $output .= '</ol>';// close pre tag just for pretty print.
                return $output;
            }

            Both approaches will have the same result.

            Let me know if you need my help at some point.

            Cheers!
            José

          • Mika

            Hey José

            Thanks a lot for your reply!

            So inherently there is not any major performance difference if I use the ob_start / ob_get_clean() and assign the code to variables?

            Also, please go ahead and activate the my code on your local environment. Then try to add a shortcode to the content page and the afterward to a text widget. You will get an error.

            It is an error that occur because the transiet filer function is called twice when the shortcode is used twice.

            How can I fix that issue? :slight_smile:

            Sincerely,
            Mika

          • Jose

            Mika,

            So inherently there is not any major performance difference if I use the ob_start / ob_get_clean() and assign the code to variables?

            Correct. Theoretically, buffering the output until it is ready to send will make a better use of network packets and provide an improvement in that regard, while in the downside it will store the buffer in memory leading to a higher use of your server's RAM.
            This could apply if we use it on a big scale. In this case, there is no impact.

            It is an error that occur because the transiet filer function is called twice when the shortcode is used twice.

            I didn't notice this when I read your code the first time, but there is a major mistake in deed. You are nesting the method definition rocapress_change_feed_cache_interval() inside another function.
            You should keep the call to add_filter() inside the function, but move the function declaration outside. Otherwise, it will try to declare the same function every time you fetch a feed.

            Change this:

            function rp_fetch_feed() {
            
                // change the default feed cache recreation period to 1 minute
                add_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
                function rocapress_change_feed_cache_interval( $seconds ) {
                // remove the custom feed internal
                remove_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
                return 60;
                }
            ...

            Into this:

            function rocapress_change_feed_cache_interval( $seconds ) {
                // remove the custom feed internal
                remove_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
                return 60;
            }
            
            function rp_fetch_feed() {
            
                // change the default feed cache recreation period to 1 minute
                add_filter( 'wp_feed_cache_transient_lifetime' , 'rocapress_change_feed_cache_interval' );
            ...

            That should do the trick.
            You may notice a warning message regarding the Cache class handler being called statically if you run in strict mode. This is a know issue in WP core and it is reported in trac. You should just ignore it. :slight_smile:

            Cheers,
            Jose

          • Mika

            Amazing, Jose!

            Thanks for the explanation!

            You are a hero! It works perfectly when I moved the function outside of the other function, and yes, that makes completely sense ( I don't know why I didn't catch that one ) :smiley:

            Last thing, is the SimplePie cache "true" ( enabled ) by default, or is it "false" ( disabled ) by default?

            I am extremely grateful for all your help Jose!!! Words can't describe my thankfulness!

            Sincerely,
            Mika

  • Mika

    Hey @Jose

    I hope you are doing good. I thought of extending this thread a bit since it is related to the topic at hand, and other developers might find it helpful.

    As I am developing my plugin, I want to add some parameters to my shortcode. For example, I would like to add a parameter named "posts", where the user for example can add 4, like this: [myshortcode ="4"]. - I know how to do this.

    Nevertheless, in the top of my shortcode function and my dashboard function, I have included the fetch feed function like this:

    /* Call the rdbv_fetch_feed() function, which fetches the feed from Amino.dk */
        $feed_items = rdbv_fetch_feed();
       // add shortcode code

    However, this also means that the parameter for how many feeds I am getting already has been set in the rdbv_fetch_feed() function.

    So my question is, how can I modify info from the rdbv_fetch_feed() in my_shortcode() function? Is that done through a filter or is there a smarter way of doing this?

    Sincerely,
    Mika

  • Jose

    Mika,
    Not sure if I get what you mean.
    If you want to use the parameter 'posts' from the following shortcode:
    [myshortcode posts = "4"]
    then you need to do something like the following :

    function rocapress_shortcode( $atts ) {
    	$atts = shortcode_atts(
    		array(
    			'posts' => 10,// Default value
    		), $atts);
    
    	$feed_items = rdbv_fetch_feed($atts['posts']);
    ...
    }

    So, your function rdbv_fetch_feed() needs to be modified to receive the parameter an do something with it rdbv_fetch_feed($posts).

    Make sense?

    Cheers,
    José

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.