Block AhrefsBot with robots.txt if you want a polite request; block it with Cloudflare or a WAF if you want enforcement. Robots.txt is simple, free, and usually respected by Ahrefs. Cloudflare and WAF bot rules are stronger because they stop requests before they hit your server.
TLDR: Start with robots.txt, then add a Cloudflare WAF rule if AhrefsBot keeps eating bandwidth or hitting sensitive paths. For example, one content site that saw 18,400 AhrefsBot requests in 24 hours cut crawler traffic by about 96% after adding a Cloudflare user agent block. If you use Ahrefs for your own SEO work, block only the bot types or paths you do not need.
Why Site Owners Block AhrefsBot
AhrefsBot is the web crawler used by Ahrefs to collect data for backlink indexes, SEO reports, keyword tools, and competitor research. It is not malware. It is not usually trying to break your site. Still, it can be noisy.
On small VPS hosting, heavy crawling can mean slower pages, higher CPU usage, more log clutter, and wasted bandwidth. On ecommerce sites, bots may hit filters, search URLs, pagination, and faceted category pages. That can create thousands of useless requests. Honestly, it feels like paying for server resources so someone else can build a database.
There is also a privacy angle. Some publishers do not want third-party SEO tools mapping every backlink, folder, redirect, or staging mistake. Blocking AhrefsBot will not hide public pages from the internet, but it can reduce how much Ahrefs learns directly from your site.
Option 1: Blocking AhrefsBot with Robots.txt
The easiest method is to add a rule to your site’s robots.txt file. This file sits at the root of your domain, such as https://example.com/robots.txt, and tells crawlers which URLs they should not crawl.
To block AhrefsBot from the whole site, use:
User-agent: AhrefsBot
Disallow: /
To block only certain folders, use:
User-agent: AhrefsBot
Disallow: /private/
Disallow: /search/
Disallow: /cart/
This is clean and fast. It takes less than a minute on most sites. You can update the file, test the URL, and wait for AhrefsBot to fetch the new instruction.
Pros of robots.txt:
- Easy to add. No firewall setup needed.
- Low risk. It does not block human visitors.
- Transparent. Anyone can see the rule and crawler policy.
- Good for broad crawling control. Great for low-drama bot management.
Cons of robots.txt:
- It is voluntary. Bad bots can ignore it.
- It does not stop requests instantly. Crawlers may revisit the file first.
- It does not protect server resources from all bot traffic.
- It can expose paths. Listing private folders can reveal areas you care about.
Ahrefs states that it respects robots.txt, so this is the right first step for many sites. The catch is that robots.txt is a request, not a lock. If you need hard blocking, use a firewall rule.
Option 2: Blocking AhrefsBot with Cloudflare
Cloudflare can block AhrefsBot before the request reaches your origin server. This is better for performance because your hosting never has to process the blocked request.
A basic Cloudflare WAF custom rule can match the user agent string. In Cloudflare, go to Security → WAF → Custom rules, then create a rule like:
lower(http.user_agent) contains "ahrefsbot"
Set the action to Block. If you are not ready to fully block it, use Managed Challenge or Log first. Logging for 24 to 48 hours is a smart move because it shows how often the bot appears and which URLs it hits.
Cloudflare is useful when:
- AhrefsBot is hitting too many URLs per day.
- Your server logs are bloated with crawler traffic.
- You want to protect expensive pages, search pages, or API routes.
- You run WooCommerce, Magento, Shopify headless, or other resource-heavy setups.
- You want proof in firewall analytics before making a firm block.
Robots.txt vs Cloudflare: Which Is Better?
Use robots.txt when you want to tell AhrefsBot not to crawl. Use Cloudflare when you want to stop it from reaching your server. That is the real difference.
| Method | Best For | Strength | Main Weakness |
|---|---|---|---|
| Robots.txt | Simple crawl control | Polite and easy | Not enforced |
| Cloudflare WAF | Hard blocking | Stops traffic early | Needs careful rule setup |
| Server firewall | Origin-level blocking | Strong | More technical |
If your site is small and AhrefsBot only visits now and then, robots.txt is enough. If you see thousands of requests per day, Cloudflare is the cleaner fix. Expect to waste time on messy logs if you skip measurement first; crawler spikes often hide inside normal traffic until you filter by user agent.
Using Other WAF and Bot-Blocking Tools
Cloudflare is not the only option. Sucuri, Fastly, Akamai, Imperva, Wordfence, NGINX rules, Apache rules, and hosting firewalls can all block by user agent, IP range, rate, or request pattern.
A server-side NGINX block may look like this:
if ($http_user_agent ~* "AhrefsBot") {
return 403;
}
For Apache, you may use:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsBot [NC]
RewriteRule .* - [F,L]
These rules work, but they still use your server. A CDN or edge WAF usually saves more resources because the request is stopped earlier.
Be Careful If You Use Ahrefs Yourself
If your marketing team uses Ahrefs, a full block may cause missing data, failed audits, or weaker rank tracking context. Ahrefs also has tools that may use separate crawlers for site audits. Before blocking everything, ask one simple question: Do we need Ahrefs to crawl our own site?
If the answer is yes, block only problem areas. For example:
/search//cart//checkout//?filter=style URLs- Internal staging or test paths
You can also rate-limit instead of block. A rate limit such as 60 requests per minute may be enough to stop traffic spikes without cutting off useful SEO data.
Recommended Setup
For most sites, use a layered setup:
- Add robots.txt rules for AhrefsBot.
- Check logs after 24 to 72 hours.
- Add a Cloudflare WAF rule if requests continue or server load stays high.
- Use path-based rules if you still want Ahrefs to crawl public content.
- Review firewall analytics monthly so you do not block useful tools by accident.
A practical default is simple: robots.txt for policy, Cloudflare for enforcement. Robots.txt tells compliant crawlers what you want. Cloudflare makes sure your server is not stuck handling traffic you never wanted in the first place.
