How to save/copy/archive a website that is going to be closed down?

This page summarizes the projects mentioned and recommended in the original post on /r/DataHoarder
Archiving crawl Spider Crawler warc

WorkOS
Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • browsertrix-crawler

    Run a high-fidelity browser-based crawler in a single Docker container

    • I'm looking for the same tool. https://github.com/webrecorder/browsertrix-crawler claims to do the job but it doesn't scale well, runs only on a single machine, doesn't support resumes, etc.

  • grab-site

    The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

    • Thanks for pointing that out! You led me to https://github.com/ArchiveTeam/grab-site which makes it so easy to grab a site by myself.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts