New to the community? Start here

[SmartCrawl Pro] Problem to create "sitemap.xml" and "Robots.txt"

Dear all,

I tried many times to make sitemap in my website by SmartCrawl Pro and it seems everything is fine but when I Run checkup, the result is something else.

I added a robot.tx file to my root web host which is beside the wp-config and … but it can not detect it.

I will be happy if you can help me with that.

Access support is active.

Thanks in advance

I will add some images to understand better about my issue

Fri, 30 Aug 2019 10:14:10

Hadi
- WPMU LOVER
- 99 pts: Hero points 62 pts: Rep. points LEVEL 3
I have attached some images about my issue

Message about robots.txt

Message about sitemap.xml

Message about sitemap.xml

Text inside of my robots.txt

Place of my sitemap.xml and robots.txt
Adam
- Support Gorilla
- 5,501 pts: Hero points 32,091 pts: Rep. points LEVEL 30
Hi Hadi

I hope you’re well today!

I tried to access your site to check it but apparently the support access is not currently active. Could you please enable it again for me?

In case it was showing it is enabled, please revoke it and grant again (in some rare cases it’s necessary for it to “catch up”:wink:. You can do it on “WPMU DEV -> Support -> Support Access” page in your site’s back-end.

As for the issue. I see that you got “sitemap.xml” file in a root folder of your site install (at least that’s how it look like on screenshots). This file shouldn’t be there because while it’s a correct location for sitemap, SmartCrawl doesn’t create file there – instead it puts the file into the /wp-content/uploads folder and adds WP rewrite. If the file physically exists in root folder of the install, it will overwrite the SmartCrawl’s one.

So, the first step would be to physically remove the sitemap.xml file from the root folder of WP install and then re-save permalinks (no need for changes, just re-save them as they are) and clear all caches on site and server, if they are there. After that, please run SmartCrawl -> Sitemap -> Crawl again and see if the sitemap gets created.

As for the robots.txt. I can access it and I see there’s a link to sitemap added there but there’s a “catch” also. I’m not sure how your site is setup (since I don’t have access to it and also I don’t have any access to installation on the server) but Google actually checks – and requires sitemap info there – the robots.txt in the root folder of the… domain.

So, if for example you got a structure similar to this on server:

domain.com (nothing here)

domain.com/site (one install here)

domain.com/something (another install here)

domain.com/something/other (yet another separate WP install here)

then for all these installs the robots.txt folder is expected at… domain.com/robots.txt.

I realize this a bit “strange”, especially that even if you don’t put physical robots.txt folder in your WP install, WordPress will server virtual one anyway, but that’s how robots work.

They would look at

domain.com/robots.txt

and

subdomain.domain.com/robots.txt

as separate robots.txt file but they won’t even look at

domain.com/subfolder/robots.txt

at all. So that’s the most likely reason for the “sitemap information not being detected” in robots.txt.

Best regards,

Adam
Hadi
- WPMU LOVER
- 99 pts: Hero points 62 pts: Rep. points LEVEL 3
Dear Adam.

I tried to access your site to check it but apparently the support access is not currently active. Could you please enable it again for me?

Done
Hadi
- WPMU LOVER
- 99 pts: Hero points 62 pts: Rep. points LEVEL 3
Dear Adam,

I can provide you the username and password of the host but How can I give it to you in private?
Hadi
- WPMU LOVER
- 99 pts: Hero points 62 pts: Rep. points LEVEL 3
Dear Adam,

Thank you very much.

With your guides, I could solve the issue.

I am wondering if I can ask any other question which is not about robots.txt and sitemap or not?

Of course, it's about SmartCrawl pro.

I will ask but Please let me know if I should publish a new ticket.

My question is about "Anchor Tags" and "Relative Anchors Pointing to Invalid IDs".

Can you please help to fix the issues?

Sometimes I think traditional ways are better than new ways ( it's wrong of course :wink: ) because I know what to do or how to do but I love WPMUdev and I should match myself to it.

Thanks in advanced
Adam
- Support Gorilla
- 5,501 pts: Hero points 32,091 pts: Rep. points LEVEL 30
Hi Hadi

Thank you for your response!

With your guides, I could solve the issue.

I’m glad to hear that and thank you for letting me know! I can see sitemap now and it looks valid so that’s a good news :slight_smile:

My question is about “Anchor Tags” and “Relative Anchors Pointing to Invalid IDs”.

Anchor Tags

An “anchor tag” is an A html tag so basically a link. This issue shows up because there were A tags detected that do not contain anchor text. It means that they are either empty or the link is only an image. This is mostly “accessibility related” – screen readers will not be able to properly “read” such links. SmartCrawl points to links to “#home” so I checked homepage and found two of them.

First one is the main logo on the page (top-left corner) that links to #home. The HTML of it is as follows (note: I masked URLs):
```
<a href="#home" class="scroll"> <img class="site_logo" src="https://yourdomain/wp-content/uploads/2019/08/logo-1.png" alt="ALT TEXT"> </a>
```
Second one is “scroll up” icon that shows in the bottom-right corner of the screen. It’s HTML is

<a href="#home" class="scroll t-center white"></a>

In both cases there’s only an image (image in first and icon in second) inside the A tag but no text. That’s why this issue is reported. It’s just these two links so it’s not a “big deal” if it comes to SEO itself but it’s very important if it comes to site accessibility (and can have bigger impact on SEO in future).

Solving this would most likely require editing the theme. You would need to find a way in the theme (in the child-theme preferably) to add some text to both these links. For example (first and second respectively):
```
<a href="#home" class="scroll"> <img class="site_logo" src="https://yourdomain/wp-content/uploads/2019/08/logo-1.png" alt="ALT TEXT"> Home</a>
```
<a href="#home" class="scroll t-center white">Scroll to top</a>

Note that I’ve used “span” element to wrap the text. I believe you don’t want to actually show any text there so this would let you then use CSS to e.g. set text color to background or make it transparent so human visitors wouldn’t see it but screen readers would still get it.

Relative Anchors Pointing to Invalid IDs

This is something you can safely ignore in this specific case. The warning shows up because some anchors (A tags) were detected with “href” attribute pointing to non-existing IDs. For example:

– this points to existing ID

<a href="#home">

– but this does not

<a href="#">

There are some links like that second one on the site but in this case they are coming from JavaScript code. JavaScript scripts actually do use that kind of structure/construction (though they don’t have to, it can be done in a different way) sometimes but to address it you’d either need to re-write those scripts on site (if they are “yours” – created by you or custom developed for you) or to get plugin/theme developers to rewrite them. I would, however, just ignore that. It won’t have any significant impact on SEO in this case.

Best regards,

Adam
Hadi
- WPMU LOVER
- 99 pts: Hero points 62 pts: Rep. points LEVEL 3
Thanks dear Adam