Ever surprise which search engine optimization bots are essentially the most blocked? This could affect the standard of the info the instruments present.
Blocking these bots will largely affect the hyperlink index of the instruments. They received’t have the ability to crawl the pages, to allow them to’t test the place these pages are linking. It doesn’t matter for site visitors estimates, key phrase rankings, high pages, and many others. These are constructed from totally different information sources.
For Ahrefs, it might additionally affect the inner hyperlinks we present and the web page historical past function that exhibits modifications to your pages over time, which you may want sooner or later. Ahrefsbot additionally powers the index for our search engine, Yep.com, so blocking Ahrefsbot means you wouldn’t present in Yep’s search outcomes.
We checked out ~140 million web sites to see how typically search engine optimization bots have been blocked. I wish to give an enormous due to our information scientist Xibeijia Guan for pulling this information.
Listed here are the highest 3 most blocked search engine optimization bots:
- MJ12bot (Majestic). Blocked by 6.49% of all web sites.
- SemrushBot. Blocked by 6.34% of all web sites.
- AhrefsBot. Blocked by 6.31% of all web sites.
We appeared on the whole variety of web sites blocking the bots. There are numerous methods to dam bots with robots.txt, and this accounts for all of them together with:
- Specific blocks, the place the bot is talked about and disallowed
- Normal blocks, the place all bots could also be blocked
- Any situations the place a directive allowed the bot, after blocking all bots
Caveats: this doesn’t embrace every other block varieties equivalent to firewalls or IP blocks.
As I discussed earlier, essentially the most blocked bot is MJ12bot from Majestic. I think there are a pair causes for this.
- They’re a distributed crawler, that means you’ll be able to’t lookup or block them by IPs, which makes them much less trusted.
- They’ve been crawling the online for longer.
- They’ve a smaller person base than extra fashionable search engine optimization instruments and due to this fact much less leverage to take away any blocks.
Listed here are essentially the most blocked search engine optimization bots:
And the overall web sites blocking search engine optimization bots:
Right here’s the information:
| Bot Identify | Depend | Proportion % | Bot Operator |
|---|---|---|---|
| MJ12bot | 9081205 | 6.49 | Majestic |
| SemrushBot | 8868486 | 6.34 | Semrush |
| AhrefsBot | 8831316 | 6.31 | Ahrefs |
| dotbot | 8569766 | 6.13 | Moz |
| BLEXBot | 8374216 | 5.99 | search engine optimization PowerSuite |
| serpstatbot | 7878935 | 5.63 | Serpstat |
| DataForSeoBot | 7872939 | 5.63 | DataForSEO |
| SemrushBot-CT | 7855400 | 5.62 | Semrush |
| Barkrowler | 7804425 | 5.58 | Babbar |
| SemrushBot-BA | 7796785 | 5.57 | Semrush |
| SemrushBot-SWA | 7789812 | 5.57 | Semrush |
| SemrushBot-SI | 7789062 | 5.57 | Semrush |
| SEOkicks | 7758904 | 5.55 | SEOkicks |
| Screaming Frog search engine optimization Spider | 7711108 | 5.51 | Screaming Frog |
| linkdexbot | 7704425 | 5.51 | LinkDex |
| DomainStatsBot | 7696944 | 5.5 | Domainstats |
| ZoomBot | 7669495 | 5.48 | SEOZoom |
| SiteCheckerBotCrawler | 7666545 | 5.48 | Sitechecker |
| Cocolyzebot | 7666233 | 5.48 | Cocolyze |
| SeobilityBot | 7664228 | 5.48 | Seobility |
| SenutoBot | 7655145 | 5.47 | Senuto |
| hypestat | 7648671 | 5.47 | HypeStat |
| online-webceo-bot | 7648444 | 5.47 | WebCEO |
| BrightEdge Crawler | 7648139 | 5.47 | BrightEdge |
| SEOlizer | 7648112 | 5.47 | SEOLizer |
It will get just a little extra difficult to investigate. For the above, we appeared on the principal robots.txt file for a web site, however each subdomain can have their very own set of directions. If we take a look at the ~461M robots.txt in whole, then essentially the most blocked search engine optimization bot is SemrushBot at 5.76%. Listed here are the highest 5:
- SemrushBot: 5.76%
- Dotbot (Moz): 5.34%
- MJ12bot (Majestic): 4.96%
- BLEXBot: 4.88%
- Ahrefsbot: 4.67%
For this measure, we’re wanting solely at instances the place a selected bot is disallowed. It doesn’t embrace any total disallow statements or instances the place solely sure bots could also be allowed. In these instances, web site homeowners went out of their technique to particularly block sure bots.
Majestic’s bot is essentially the most focused, adopted by Moz’s bot.
Listed here are essentially the most blocked search engine optimization bots by specific mentions:
Listed here are the variety of web sites explicitly blocking search engine optimization bots:
Right here’s the information:
| Bot Identify | Depend | Proportion % | Bot Operator |
|---|---|---|---|
| MJ12bot | 2000372 | 1.43 | Majestic |
| dotbot | 1402305 | 1 | Moz |
| AhrefsBot | 1350771 | 0.97 | Ahrefs |
| SemrushBot | 1285857 | 0.92 | Semrush |
| BLEXBot | 861184 | 0.62 | search engine optimization PowerSuite |
| serpstatbot | 354683 | 0.25 | Serpstat |
| DataForSeoBot | 284694 | 0.2 | DataForSEO |
| Barkrowler | 276332 | 0.2 | Babbar |
| SEOkicks | 219961 | 0.16 | SEOkicks |
| SemrushBot-CT | 211895 | 0.15 | Semrush |
| linkdexbot | 166405 | 0.12 | Linkdex |
| DomainStatsBot | 157053 | 0.11 | Domainstats |
| SemrushBot-BA | 154349 | 0.11 | Semrush |
| SemrushBot-SI | 147999 | 0.11 | Semrush |
| SemrushBot-SWA | 146261 | 0.1 | Semrush |
| ZoomBot | 125310 | 0.09 | SEOZoom |
| SiteCheckerBotCrawler | 122574 | 0.09 | Sitechecker |
| Cocolyzebot | 121737 | 0.09 | Cocolyze |
| SeobilityBot | 117558 | 0.08 | Seobility |
| Screaming Frog search engine optimization Spider | 87673 | 0.06 | Screaming Frog |
| SenutoBot | 54978 | 0.04 | Senuto |
| hypestat | 861 | 0 | HypeStat |
| SenutoBot | 54978 | 0.04 | Senuto |
| hypestat | 861 | 0 | HypeStat |
| online-webceo-bot | 659 | 0 | WebCEO |
| BrightEdge Crawler | 289 | 0 | BrightEdge |
| SEOlizer | 253 | 0 | SEOLizer |
We appeared on the high 1M websites by DR, which aligns to websites with a DR >45. Semrush is essentially the most blocked adopted by Majestic and Moz.
Right here’s the way it breaks down for every particular person bot in numerous classes of internet sites. The highest 3 are:
- Autos_and_Vehicles: 39%
- Books_and_Literature: 27%
- Real_Estate: 17%
Going by the bot requests in Cloudflare Radar, Ahrefs is by far the quickest crawler within the search engine optimization area. ~4.6x quicker than Moz and ~6.7x quicker than Semrush.
Support authors and subscribe to content
This is premium stuff. Subscribe to read the entire article.
