Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud

• apache-ultimate-bad-bot-blocker

  3 742 9.5 C

  Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders

I found Ahrefs a bit too aggressive. Thankfully they seem to honor Robots.txt so it's not much of a problem, but across the wide range of sites I'm responsible for, the Ahrefs bots really stood out as being a problem. Out of curiosity, I just double checked and the only hits from them in the logs are "GET /robots.txt" which is what I would expect.

I recently found a regularly update place for bot stuff:

https://github.com/mitchellkrogza/apache-ultimate-bad-bot-bl...

I don't think I'd call Ahrefs a "Bad Bot" but they are on that list. I'd call any bot ignoring robots.txt bad though.

• apache-ultimate-bad-bot-bl

  1 - -

I found Ahrefs a bit too aggressive. Thankfully they seem to honor Robots.txt so it's not much of a problem, but across the wide range of sites I'm responsible for, the Ahrefs bots really stood out as being a problem. Out of curiosity, I just double checked and the only hits from them in the logs are "GET /robots.txt" which is what I would expect.

I recently found a regularly update place for bot stuff:

https://github.com/mitchellkrogza/apache-ultimate-bad-bot-bl...

I don't think I'd call Ahrefs a "Bad Bot" but they are on that list. I'd call any bot ignoring robots.txt bad though.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• infinite-fake-website

  3 7 4.0 PHP

  Lure SEO spammers down a hole with no bottom

I kind of wish they hadn't. Ahrefs claims to be in the "SEO" business, which as far as I'm concerned, has helped ruin web search, and indeed, the culture of the web in general.

Also, Ahrefs bot doesn't handle some things very well. I made an "infinite web site" in PHP a while back, and used Apache mod_rewrite to send every Ahrefs request to the infinite web site PHP program: https://github.com/bediger4000/infinite-fake-website

Ahrefs bot really freaked out, unlike some professional bots like Google's, and even Yandex' bot.

Suggest a related project