How is ArchiveBox?

• browsertrix-crawler

  13 538 9.0 TypeScript

  Run a high-fidelity browser-based crawler in a single Docker container

  

If you need more advanced recursive spider/crawling ability beyond --depth=1, check out Browsertrix, Photon, or Scrapy and pipe the outputted URLs into ArchiveBox.

• Scrapy

  180 50,824 9.6 Python

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  

If you need more advanced recursive spider/crawling ability beyond --depth=1, check out Browsertrix, Photon, or Scrapy and pipe the outputted URLs into ArchiveBox.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• Photon

  3 10,501 0.0 Python

  Incredibly fast crawler designed for OSINT. (by s0md3v)

If you need more advanced recursive spider/crawling ability beyond --depth=1, check out Browsertrix, Photon, or Scrapy and pipe the outputted URLs into ArchiveBox.

• ArchiveBox

  248 19,737 9.7 Python

  🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

  

From https://archivebox.io/:

Suggest a related project