You have verified your website in Google Search Console, and you want to run a website audit with the Ahrefs Site Audit Tool, but your audit keeps failing.
It might have worked the first time, but now when you try to audit your website, you get the same error:
I had this issue with my website, each time I tried an audit, I got this message:
Let’s take a look at some troubleshooting to fix this issue. Please note, this information is checked and updated when Ahrefs update on their end.
Important Updates from Ahrefs
Troubleshooting
There are a few steps we can look at to avoid this issue, and make it that the Ahrefs Site Audit Tool can succesfully audit your website.
Update your Robots.txt file
You might need to update your Robots.txt file. This is the file that tells website crawlers, such as Google Bot, what can be crawled on your website.
Check your Robots.txt file to see if you are allowing website crawlers access to your website.
A normal Robots.txt file looks like this:
User-agent: *
Disallow:
Add the Ahrefs Crawler to your Robots.txt file
1 – Go to your Robots.txt file
If you have a WordPress website, you can find this through the Yoast Plugin. Go to Yoast SEO, click on Tools, then click on File editor. You will now see your Robots.txt file at the top.
If you don’t have WordPress, you might need to ask your web developer for help. If you use a website builder, like Wix or SquareSpace, you can find the Robots.txt file in the admin section of you website.
A Robots.txt file should look something like this: (WordPress website)
User-agent: *
Disallow: /wp-admin/
The * on the User-agent: means that any crawler has access to this website – to be sure, we can add the Ahrefs Crawler.
The Disallow: section can list pages and links that you don’t want a crawler to look at. For example, if you have a hidden page, or pages that you do not want to appear on a search engine, you can add them with Disallow: – one row at a at time.
2 – Add the Ahrefs Crawler
This is the Ahrefs Crawler:
User-agent: AhrefsSiteAudit
Allow: /
To add it to your Robots.txt, add a new line, and add the code as above:
User-agent: *
Disallow: /wp-admin/
User-agent: AhrefsSiteAudit
Allow: /
3 – Save changes and try your Audit again
Save the changes to your Robots.txt file and refresh the Ahrefs Site Audit Tool – now try your audit again.
Still no luck? Try the next solution.
Test your Robots.txt file can be crawled by Ahrefs
Ahrefs have a useful tool that you can use to check if they can crawl your website.
Go to ahrefs.com/robot/site-audit and enter your full website URL. This will tell you if your website can be crawled.
How long does it take for Google bot to crawl your robots.txt file?
Google bot usually crawls your robots.txt file every 24 to 36 hours.
To learn more about Robots and Google bot, see the resource ‘verifying Googlebot and other Google crawlers‘ on the Google Search Central site.
Still getting the ‘This website can’t be crawled.’ error?
Try the next solution.
Whitelist the Ahrefs IP Address Range
Ahrefs uses an ‘IP address range’ – this means that they use different IP addresses to crawl your website.
In some cases, your hosting provider my have blocked these IP addresses, meaning that the Ahrefs Crawler can’t audit your website.
You will need to send a support ticket, or contact your hosting provider and send them the IP range that Ahrefs use.
Cloudflare Users
You can check the instructions for whitelisting IP lists on Cloudflare with the dev documentation.
For further information about whitelisting IPs in Cloudflare, see ‘How to whitelist Ahrefs IPs in Cloudflare?‘.
If Cloudflare does not accept /26 and /27 ranges, please use the individual IPs listed below. It is important to note that both IP ranges and individual IPs (listed above) must be entered for our IPs to be fully whitelisted.
These might be subject to change, so please refer to this link to see the most updated information – https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges
HTTP server returned error 403: “Forbidden”
If you get the error ‘403: forbidden’ – this means that the Ahrefs crawler is blocked by your website’s firewall or on-site plugin. To troubleshoot this error, whitelist the Ahrefs IP addresses, found above.
It is always recommended that you ask your hosting provider to whitelist any IP Addresses, as this usually is done through the .htaccess file through your FTP server.
If you use a plugin on your website for security, there should be options in the plugin to whitelist IPs.
Error 406 Not Acceptable: Firewall
If you keep getting an error message that says ‘Error 406 Not Acceptable: Firewall’, it could be an issue with the configuration of your webserver, the firewall managed by your hosting provider, or the protection of your CDN.
There is nothing that Ahrefs can do to fix this, as you will have to contact your hosting provider.
Here is a template you can use to email your hosting provider:
I own the domain
Please see more details about the crawler here: https://ahrefs.com/robot
Thank you very much.
Ahref Crawler Bots
Ahrefs use 2 bots to crawl websites, their marketing bot and their SEO bot. The SEO is bot is used for their Site Audit Tool, that looks through the health of a website.
Where as the marketing bot is used more frequently to get an overview for backlinks and domain authority, along with the Ahrefs keyword and link intersect tools.
”AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill our database with new links and check the status of the previously found ones to provide the most comprehensive and up-to-the-minute data to our users.” – Ahrefs
Did you know, the AhrefsBot is used to power the new ‘Yep’ search engine?
Below are details for both bots, but you really only need to update your robots.txt file for the ‘AhrefsSiteAudit’ bot.
AhrefsSiteAudit Bot Details
| Version | 6.1 |
|---|---|
| Bot type | Good (Identifies itself, has an official moniker) |
| Category | SEO |
| Obeys robots.txt | Yes by default (website owners can request to disobey robots.txt on their sites) |
| Obeys crawl delay | Yes by default (website owners can request to disobey crawl delay on their sites) |
| Desktop user-agent string | Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit) |
| Mobile user-agent string | Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.5359.128 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit) |
| Reverse DNS suffix | ahrefs.com |
| IP address range |
AhrefsBot Details
| Version | 7.0 |
|---|---|
| Bot type | Good (Identifies itself, has an official moniker) |
| Category | Marketing |
| Obeys robots.txt | Yes |
| Obeys crawl delay | Yes |
| User-agent string | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) |
| Reverse DNS suffix | ahrefs.com |
| IP address range |
FAQs
Details updated as of March 2025.
Did that fix your issue? I hope so. There are no other troubleshooting solutions from Ahrefs at the moment, but it seems that Whitelisting the IP Address Range is the most common fix.
Let me know how you get on, and share this troubleshooting guide with your friends and colleagues ⬇
8 comments
Hi Phil, I have two relevant redirected websites pointing to my main website. For some reason Ahrefs are showing “Fetching robots.txt took too long” and “The HTTP server returned error 403: “Forbidden”. This request was likely filtered by server configuration”
I have been back and forward between GoDaddy who I purchased the expired domains from – which have succesfully been redirected to my main website – and IONOS who host my WordPress managed main website. They both say the issue is not with them.
I have also been in touch with Ahrefs who have advised that the IPs should be whitelisted – HOWEVER – as I never had access to the original domains before they expired how on earth do I do that?
Main website: https://superbsalt.com
Redirected site: http://sundayspentcooking.com
Redirected site: http://saffronpudding.com
Any advice greatly welcomed.
Hello Chris, thanks for your comment.
As your domains for http://sundayspentcooking.com and http://saffronpudding.com are redirected to your main website, https://superbsalt.com/ – you will always get this error when trying to crawl with Ahrefs.
These redirects are done on the server – so both domains are no longer acting as a separate website. Meaning you won’t be able to crawl those websites when you have a redirect in place.
If you want to crawl http://sundayspentcooking.com and http://saffronpudding.com with Ahrefs, you will need to disable the redirects, then try the bot again.
Ahrefs uses the robots.txt file, and when you set a redirect on the server (to point one domain to another) those old domains will no longer have an accessible robots.txt file, and can’t be crawled.
I hope this helps.
Phil
I’ll right away grasp your rss feed as I can not in finding your email subscription link or newsletter service. Do you have any? Please let me understand in order that I may subscribe. Thanks.
Hello, I do not provide a newsletter, but you can find my direct RSS feed here.
My website still do not crawl able. Can you tell me the issue.
Hello, on your website patracorp.com, you have not included the Ahrefs bit in your Robots.txt file. See Add the Ahrefs Crawler to your Robots.txt file. If this still does not work, it could be that your hosting provider has whitelisted Ahrefs, and you will need to contact them, see Whitelist the Ahrefs IP Address Range.
I hope that helps.
Lots of thanks for sharing this informative post. I faced same problem and now it is solved by following your guidelines.
Thank you Mahadi, I am glad you found it useful 👍
Comments are closed.