知识库
Semrush Toolkits
SEO
Site Audit
Why do working pages on my website appear as broken?

Why do working pages on my website appear as broken?

Why do working pages on my website appear as broken? image 1

There may be a time where Site Audit may show that some of your internal pages are reported as broken but are actually still fully up-and-running. While this does not happen often, it can be confusing for some users.

Generally, when this happens it's because of a false-positive. The three most common reasons for a false-positive reading are:

  • Our Site Audit crawler could have been blocked for some pages in robots.txt or by noindex tags

    Why do working pages on my website appear as broken? image 2
  • Hosting providers might block Semrush bots as they believe it is a DDoS attack (massive amount of hits during a short period of time)
  • At the moment of the campaign re-crawl, the domain could not be resolved by DNS

If you believe it’s happening because of a crawler issue you can learn how to troubleshoot your robots.txt in this article

You can also lower the crawl speed in order to avoid a large number of hits on your pages at one time. That is why you could see this page as a working one, which is right, and our bot was unable to do so and reported the false-positive result.

Site Audit crawler settings

常见问题 展示更多
手册 展示更多
工作流 展示更多