Importing large number of RSS feeds

Hello,
I've developed a website in which i want to display post's from RSS feeds from other websites. i've used wordpress ofc and after lots of trial and error with RSS importers i found that Multi-Importer plugin does exactly what i need (no autoblogged doesnt fit my needs).
My problem is that right now im importing feeds from 130 RSS feeds, and the server runs out of memory and website is lagging like hell. I moved the website into a new VPS with 4 cores and 8GB of RAM, and each time it checks for new RSS it takes up all the RAM and every 6 hours it craches the mySQL server.
My plan is to make a website with constand flow of news gathered from other websites, but im getting fraustrated because it has made my website very slow.
Im considering now to move this website into a dedicated server with 16GB of RAM or more, if it is to solve my issue because i want to have a total of 300-500 RSS feeds to run constantly (every 10min is the minimum in multi importer).
Any ideas or suggestions how to improve this project?
Thank you in advance for your time.

  • Michael Bissett

    Hey @Theodor, hope you're doing well this evening! :slight_smile:

    In regards to this:

    My problem is that right now im importing feeds from 130 RSS feeds, and the server runs out of memory and website is lagging like hell.

    Given the last part of this statement:

    because i want to have a total of 300-500 RSS feeds to run constantly (every 10min is the minimum in multi importer)

    Are you trying to import these feeds all at once, every 10 minutes? Or am I not following what you're saying here?

    If you are trying to import that many feeds at once, I'd like to ask... is it necessary to have them update that often?

    That's an awful lot of feeds to handle, and it may be best to split up the load (i.e. split up the processing of feeds into separate batches, like 100 every 30-45 minutes), so that your server doesn't crash on you (especially considering the amount of feeds that you're aiming to import).

    Please advise,
    Michael

  • Theodor Dimitriou

    Yes you have understand correctly Michael,
    The purpose of the website is to have a constand flow of news, and to achieve that you have to refresh the feeds often. Even if not every 10 mins lets say every 20mins.
    How do i split the RSS feeds into separate batches and make them run in different times in a rotating order?
    Yes multi importer plugin tries to import all feeds at once unfortunately. It doesnt have any option to CRON per feed or per bunch of feeds.
    Any suggestions?

  • Ash

    Hello @Theodor

    I hope you are well today.

    A dedicated server with 16GB ram should be able to handle this load, but it still depends.

    TBH, I have never used Multi Importer plugin myself, but this is not useful if they don't use cron. If you run 300 - 500 feeds to process at a time, there are still possibilities to crash the server.

    What I would recommend to hire a developer to customize the plugin to run on cron, so that it proceeds through the feeds in a certain interval, this way the load will be split into several segment.

    Cheers
    Ash

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.