robots-txt

Open-source projects categorized as robots-txt
Edit details
Language: + Python + TypeScript + C# + C++ + HTML + PHP
Topics: Crawler Spider Web Crawling Python SEO
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Top 11 robots-txt Open-Source Projects

  • advertools

    advertools - online marketing productivity and analysis tools

  • robots

    NuxtJS module for robots.txt

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • Infinity Crawler

    A simple but powerful web crawler library for .NET

  • gflare-tk

    Open-Source Python Based SEO Web Crawler

  • the-great-gpt-firewall

    🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

    • Project mention: The Great GPT Firewall | news.ycombinator.com | 2024-02-14

  • librengine

    Privacy Web Search Engine (not meta, own crawler)

  • ai-training-opt-out

    Known tags and settings suggested to opt out of having your content used for AI training.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • Robots Exclusion Tools

    A "robots.txt" parsing and querying library for .NET

  • robots-txt

    Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

  • MicroFrontier

    A lightweight crawler frontier implementation in TypeScript using Redis.

  • AI-Data-Guard

    AI Data Guard

    • Project mention: AI-Data-Guard: Robots.txt Analysis for GPT Bots | news.ycombinator.com | 2023-09-14
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

robots-txt related posts

Index

What are some of the best open-source robots-txt projects? This list will help you:

Project Stars
1 advertools 1,058
2 robots 386
3 Infinity Crawler 238
4 gflare-tk 140
5 the-great-gpt-firewall 67
6 librengine 64
7 ai-training-opt-out 51
8 Robots Exclusion Tools 27
9 robots-txt 8
10 MicroFrontier 7
11 AI-Data-Guard 3

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com