nginx-ultimate-bad-bot-blocker VS Global-List-Facebook-Groups-Keyword-Moderation-Alerts

I manage a network of several hundred websites. About 2 years ago we started blocking bot/crawler user-agents. Some websites see thousands of these requests every day, skewing analytics and eating bandwidth.

Tools like this help: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...

I generally prefer to go up a level. We install ultimate bad bot blocker at the nginx level for apps running on servers or in K8S. For Heroku you can do this with foremen or docker. The advantage of this is that the ‘no’ happens faster than it does when making it all the way down to rack. RackAttack might let a door to door salesperson say “hi,I’m here to talk to you about solar” before slamming the door. Nginx let’s the same salesperson barely say “hi ..” before slamming the door in their face.

the subdomain example nginx config from NextCloud with a few changes like TLSv1.3 only, geoblocking and some bot blocking from here and a few other security tweaks. The nginx file is basically a server block as an include separate from the main nginx.conf. The configuration gets an A+ in security from NextCloud's checker and others.

Use nginx anti-bot[1] or product such as Fastly or Cloudflare anti-bot feature, which blocks content scrapping and only allows specific bots using rules or AI. Another option is to block VPN, cloud and hosting companies' ASNs. There is no benefit for you allowing someone to scrap your content apart from Bing and Google. The rest of the bots can die, or you will lose your content to scrappers. So finally, put it behind a paywall if possible. Of course, the paywall only works if you have good content or buying ads to promote it. Good luck, mate.

[1]https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...

If you want to go full-shield mode you can use this solution on the server https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker and/or Cloudflare rules. Keep in mind that having both of this tools at the same time will require some deep tech work from you and will not save you from all bots.

I used the ultimate Nginx bad bot blocker on a couple of my side projects, and it is a pretty good project https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo... . Apart from the Cloudflare offers UA blocking and AI driven bot management too. Most of these bots are for content scrapping and then creating search spam results. I am a one-person show, and it hurts both financial and resources wise on severs. So I block them.

I recommend https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker to block unwanted bots

Same experience here, one spammer DOSing my site, always with similar patterns. I'm already behind Cloudflare and use https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo... but I still have to identify and block them manually, which is annoying and a waste of time. Some IP-range blocks affected users so it's not fine grained enough. I wish I could identify, expose and sue them into oblivion.

Check this out https://youtu.be/ES2yJ7_8fUk and https://github.com/mitchellkrogza/Global-List-Facebook-Groups-Keyword-Moderation-Alerts works for pages and groups