crawlers

Open-source projects categorized as crawlers
Edit details
Language: + TypeScript + HTML + R + Go + C + Python
Topics: Crawler Scraper Crawling Docker Python
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App
With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
surveyjs.io
featured

Top 6 crawler Open-Source Projects

  • isbot

    🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string (by omrilotan)

    • Project mention: Lazy loading images upon intersection in Angular | dev.to | 2023-08-01

    So back to the original problem, how do we make the final image available for search bots? We can only be explicit on the server platform as to which user agents get to see the original image. Let's start with a method that returns true for a list of bots we think we want to target. (Here is a thorough list I found on the web, funny enough I can't find twitter bot there, can you?)

  • flathunter

    A bot to help people with their rental real-estate search. 🏠🤖

  • SurveyJS

    Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

    SurveyJS logo

  • Rcrawler

    An R web crawler and scraper

  • seonaut

    Open source SEO auditing tool.

  • wget-lua

    Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

  • sneakpeek

    Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis (by flulemon)

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

crawlers related posts

  • Is fighting to stay in Berlin really worth it?

    1 project | /r/berlin | 11 May 2023

  • Sneakpeek is a framework that helps to quickly and conveniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

    1 project | /r/CKsTechNews | 30 Apr 2023

  • Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis (useful for ChatGPT plugins)

    1 project | /r/aipromptprogramming | 29 Apr 2023

  • Probleme beim Aufsetzen von Python-Bot (pipenv)

    1 project | /r/de_EDV | 20 Apr 2023

  • Apartment search is really taking a toll on us, even though we have a very good profile as renters. Time is running out with our current lease and we are desperate.

    1 project | /r/berlin | 26 Mar 2023

  • I have tracked every change of popular search results on ImmoScout24 for 3,5 years (since 2019)

    1 project | /r/berlin | 15 Dec 2022

  • Giving up on WG-Gesucht

    1 project | /r/berlin | 6 Sep 2022
  • A note from our sponsor - SurveyJS
    surveyjs.io | 29 May 2024
    With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js. Learn more →

Index

What are some of the best open-source crawler projects? This list will help you:

Project Stars
1 isbot 843
2 flathunter 787
3 Rcrawler 344
4 seonaut 160
5 wget-lua 83
6 sneakpeek 36

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com