Looking for open source software to scrape webpages but also make them searchable with a webui. (locally hosted)

• awesome-datahoarding

  6 1,010 4.9

  List of data-hoarding related tools

You might also be interested in this list, those alternatives listed are really great and better, some support the WARC format (that my program doesn't).

• Collect

  1 75 0.0 TypeScript

  A server to collect & archive websites that also supports video downloads (by xarantolus)

I created Collect a few years ago and still use it today.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• grab-site

  30 1,261 3.8 Python

  The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

  

I use grab-site to crawl website and pack it into warc archive and then feed this archive into pywb

• ArchiveBox

  248 19,790 9.8 Python

  🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

  

I landed on an opensource project called Archivebox. Its pretty amazing (basically like a locally hosted wayback machine and crawler. https://github.com/ArchiveBox/ArchiveBox It also captures different ways to ensure data integrity and can schedule! Thanks everyone for your input and apps for me to research!

Suggest a related project