go-sstables
dgraph
go-sstables | dgraph | |
---|---|---|
4 | 34 | |
255 | 20,092 | |
- | 0.4% | |
4.0 | 8.8 | |
2 months ago | 5 days ago | |
Go | Go | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
go-sstables
- GitHub - thomasjungblut/go-sstables: Go library for protobuf compatible sstables, a skiplist, a recordio format and other database building blocks like a write-ahead log. Ships now with an embedded key-value store.
-
I want to dive into how to make search engines
I've never worked on a project that encompasses as many computer science algorithms as a search engine. There are a lot of topics you can lookup in "Information Storage and Retrieval":
- Tries (patricia, radix, etc...)
- Trees (b-trees, b+trees, merkle trees, log-structured merge-tree, etc..)
- Consensus (raft, paxos, etc..)
- Block storage (disk block size optimizations, mmap files, delta storage, etc..)
- Probabilistic filters (hyperloloog, bloom filters, etc...)
- Binary Search (sstables, sorted inverted indexes, roaring bitmaps)
- Ranking (pagerank, tf/idf, bm25, etc...)
- NLP (stemming, POS tagging, subject identification, sentiment analysis etc...)
- HTML (document parsing/lexing)
- Images (exif extraction, removal, resizing / proxying, etc...)
- Queues (SQS, NATS, Apollo, etc...)
- Clustering (k-means, density, hierarchical, gaussian distributions, etc...)
- Rate limiting (leaky bucket, windowed, etc...)
- Compression
- Applied linear algebra
- Text processing (unicode-normalization, slugify, sanitation, lossless and lossy hashing like metaphone and document fingerprinting)
- etc...
I'm sure there is plenty more I've missed. There are lots of generic structures involved like hashes, linked-lists, skip-lists, heaps and priority queues and this is just to get 2000's level basic tech.
- https://github.com/quickwit-oss/tantivy
- https://github.com/valeriansaliou/sonic
- https://github.com/mosuka/phalanx
- https://github.com/meilisearch/MeiliSearch
- https://github.com/blevesearch/bleve
- https://github.com/thomasjungblut/go-sstables
A lot of people new to this space mistakenly think you can just throw elastic search or postgres fulltext search in front of terabytes of records and have something decent. The problem is that search with good rankings often requires custom storage so calculations can be sharded among multiple nodes and you can do layered ranking without passing huge blobs of results between systems.
-
What's the big deal about key-value databases like FoundationDB ands RocksDB?
I highly recommend people comfortable with Go checkout the building blocks at https://github.com/thomasjungblut/go-sstables
This codebase shows how SSTables, WAL, memtables, skiplists, segment files, and plenty of other storage engine components work in a digestible way. Includes a demo database showing how it all comes together.
- Understanding LSM Trees: What Powers Write-Heavy Databases
dgraph
- DGraph – GraphQL Database
-
How to choose the right type of database
Dgraph: A distributed and scalable graph database known for high performance. It's a good fit for large-scale graph processing, offering a GraphQL-like query language and gRPC API support.
- Is Dgraph dead? (should I continue using it)
-
Database Review: Top Five Missing Features from Database APIs
Dgraph (GraphQL, DQL)
-
Learning Graph Database data design & data modeling
Have you tried dgraph.io?
-
Getting Started with Serverless Edge - Exploring the Options
DGraph – A distributed GraphQL database with a graph backend.
-
Fluree DB - A datomic like database that I just discovered
How does it compare to, say grakn (renamed https://vaticle.com/, I think?), or draph (https://dgraph.io/), or Ontotext's GraphDB (https://www.ontotext.com/products/graphdb/), or Datomic?
-
GKE with Consul Service Mesh
Consul Connect service mesh has a higher memory footprint, so on a small cluster with e5-medium nodes (2 vCPUs, 4 GB memory), you will only be able to support a maximum of 6 side-car proxies. In order to get an application like Dgraph working, which will have 6 nodes (3 Dgraph Alpha pods and 3 Dgraph Zero pods) for high availability along with at least one client, a larger footprint with more robust Kubernetes worker nodes were required.
- Show HN: We have built a benchmark platform for graph databases
- What's the big deal about key-value databases like FoundationDB ands RocksDB?
What are some alternatives?
search-engines - Reviewing alternative search engines
cockroach - CockroachDB - the open source, cloud-native distributed SQL database.
phalanx - Phalanx is a cloud-native distributed search engine that provides endpoints through gRPC and traditional RESTful API.
Hasura - Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.
pytai - Kaitai Struct: Visualizer and Hex Viewer GUI in Python
spicedb - Open Source, Google Zanzibar-inspired permissions database to enable fine-grained access control for customer applications
hse - HSE: Heterogeneous-memory storage engine
tidb - TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://tidbcloud.com/free-trial
mitta-screenshot - Mitta's Chrome extension for saving the current view of a website.
TinyGo - Go compiler for small places. Microcontrollers, WebAssembly (WASM/WASI), and command-line tools. Based on LLVM.
search-lib - A library of classes which can be used to build a search engine.
go-mysql - a powerful mysql toolset with Go