SmartCrawl timing out and missing URLs in sitemap

The site includes over 4 000 of BuddyPress Members and thousands of properly listings that are created by the a dedicated plugin (IDX). The SmartCrawl seo crawl is timing out and only includes around 200 URLs in the sitemap. There's some realtors but no property listings.

Those listings are not custom posts. They seem to be "virtual pages". Is there any way to exclude them from crawl (e.g. by "url pattern")? Or - how to fix that timeout?

  • Huberson

    Hello xmginc
    According to SmartCrawl developer support for URL exclusion should be included in 2.1.1 release. But it will only work with exact path/url in that version. Until subsequent release that might include pattern matching.

    Regarding the timeout issue, since the site is live some advanced troubleshooting can't be performed there. I would suggest setting up a clone/staging of the live site for testing.
    Once you have that setup you can send over the following info to us using our contact form and let us know so we can check the issue:

    Subject: "Attn: Huberson Dorvilus"
    -WordPress admin username
    -WordPress admin password
    -login url
    -FTP credentials (host/username/password)

    -link back to this thread for reference

    -any other relevant urls

    Regards,
    Huberson

Thank NAME, for their help.

Let NAME know exactly why they deserved these points.

Gift a custom amount of points.