Autoblog Suggestion to trace the full post links from the RSS feed

Can we have the feature in the Autoblog plugin that it will trace the full post links from the RSS feed, crawl those posts and fetch the full content of the posts even if the feed only has the excerpt?

  • Adam Czajczyk

    Hello jai

    I hope you're well today and thank you for your question!

    I realize that what I'm about to say may not be an expected answer and I'm sorry in advance but I wouldn't expect such feature any time soon. There's a couple of problems with this:

    1. what you're suggesting is called "scrapping" and there are some "technical difficulties" with this as it requires actually reading and parsing the source of a given page; such "parsing" is never "universal" as it's really depending on the site's DOM structure which is different with different themes and usually is also affected by some additional plugins and so on. So, even if we provided such "tool" that would fetch and be able to parse such data, it would still require quite a lot of "manual" work for you to actually "teach" the plugin which parts of the fetched page is the content to be scrapped and which not; that would have to be done separately for each feed (sometimes even multiple times for the same feed if e.g. page templates for different articles would be different).

    2. it goes a bit into a "shady areas" of Internet because scrapping in most countries means actually violation of Intellectual Property Rights - if it's an unauthorized use of content. Of course, you could obtain permission from the site that you are fetching content from but then if they can give you permission they can usually also provide you with tools (like separate feed or some API or some closed download area) that would let you download those articles. On the other hand, in some cases providing a "tool that helps violate Intellectual Property Rights" could possibly get us into serious troubles too.

    That being said, please don't take my word on it as an "official company statement". It's more of my personal remark just to show the scale of the difficulty with such implementation. Let's just keep that ticket open here as a feature request and see if more Members would support the idea.

    If it gets more votes we'll look into possible solutions (taking both "technical" and "legal" aspects into account) and our developers will consider implementing this.

    Best regards,
    Adam

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.