heritrix3
Elasticsearch
| heritrix3 | Elasticsearch | |
|---|---|---|
| 6 | 91 | |
| 2,700 | 67,789 | |
| 1.4% | 0.8% | |
| 6.2 | 10.0 | |
| 2 days ago | 6 days ago | |
| Java | Java | |
| GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
heritrix3
-
WARC'in the Crawler
If anyone's interested in web crawling technology, check out Heretrix [1], been around since 2004 and while not the most performant it has incorporated many responsible disciplines in the design and as this article pointed out, WARC format.
1. https://heritrix.readthedocs.io
-
Why isn't the internet more fun and weird?
I've considered running my own IA (https://github.com/internetarchive/heritrix3), which only archives my chrome history, which is stored in a local sqlite db.
I haven't had the time to figure out the details of how the pieces should be glued together.
-
Is there a way to archive groups of webpages similarly to how web archive does it?
You can go all the way and use the same tools as the Internet Archive: https://github.com/internetarchive/heritrix3
- Heritrix: Internet Archive's extensible, web-scale, archival-quality web crawler
- Best Http client for web scraping
-
The Internet Is Rotting
By the way, the technical side of this is very interesting. If you look at the tools mentioned (the wayback machine, but also perma.cc and other archival solutions), almost all of them rely on a single semi-modern tech stack that produces WARCs (web archives - ISO - ISO 28500:2017 https://iipc.github.io/warc-specifications/specifications/wa...).
The main crawler still seems to be heritrix3 (https://github.com/internetarchive/heritrix3), but there's a great little ecosystem with tools such as webrecorder and warcprox.
Still, I've read through the code of these tools and am feeling that they are failing in the face of the modern web with single page apps, mobile phone apps and walled gardens. Even newer iterations with browser automation are getting increasingly throttled and blocked and excluded from walled gardens.
Perhaps the time has come for a coordinated, decentralized but omnipresent approach to archival.
Elasticsearch
-
Elasticsearch Version 9
You could check out their GitHub and see what is going on https://github.com/elastic/elasticsearch/issues
- One .gitignore to rule them all
-
Who's hiring developer advocates? (October 2023)
Link to GitHub -->
-
Do we think about vector dbs wrong?
I believe the 1024 limit has been upped in recent versions of Elasticsearch
https://github.com/elastic/elasticsearch/issues/92458
-
Elasticsearch VS openobserve - a user suggested alternative
2 projects | 30 Aug 2023
- A dedicated Elasticsearch query language (ES|QL)
- Fleet datastreams: custom index templates
-
Integrating Elasticsearch with Node.js Applications
Elasticsearch is written in Java and its source code is available on Github.
-
Murmur3 hash plugin for nested objects?
I don't think the murmur3 hash implementation has changed since it was added as the default in version 2.0 (see the [changes](https://github.com/elastic/elasticsearch/commits/main/server/src/main/java/org/elasticsearch/cluster/routing/Murmur3HashFunction.java)). The plugin itself has seen [more changes](https://github.com/elastic/elasticsearch/commits/main/plugins/mapper-murmur3) but that's IMO because of internals and not visible changes in the calculations.
-
Mongo or Mysql for 10tb of JSON documents, I'm questioning my previous choice.
Mysql is not as open source as postgres (long story). And you can see how open elasticsearch is by just having access to the bugs database https://github.com/elastic/elasticsearch/issue
What are some alternatives?
tubeup - Use yt-dlp to download video and upload to the Internet Archive with metadata.
OpenSearch - 🔎 Open source distributed and RESTful search engine.
brozzler - brozzler - distributed browser-based web crawler
Apache Superset - Apache Superset is a Data Visualization and Data Exploration Platform [Moved to: https://github.com/apache/superset]
upvotocracy-ui-ssr - Free speech Reddit clone
bleve - A modern text/numeric/geo-spatial/vector indexing library for go
pgvector - Open-source vector similarity search for Postgres
Whoosh
MeiliSearch - A lightning-fast search API that fits effortlessly into your apps, websites, and workflow
elasticsearch-dsl-py - High level Python client for Elasticsearch
Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company :yum:
Milvus - A cloud-native vector database, storage for next generation AI applications