ldbc_snb_datagen_spark
hash-db
ldbc_snb_datagen_spark | hash-db | |
---|---|---|
5 | 5 | |
165 | 50 | |
1.8% | - | |
3.7 | 0.0 | |
15 days ago | over 1 year ago | |
Java | Python | |
Apache License 2.0 | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ldbc_snb_datagen_spark
-
Benchgraph Backstory: The Untapped Potential
Because of the size, complexity, and feedback from the community, we decided to add a larger dataset. So the next dataset should be large, more complex, and recognizable. The choice was easy here; the industry-leading benchmark group Linked Data Benchmark Council (LDBC), which Memgraph is a part of, has open-sourced the datasets for benchmarking. The exact dataset is the social network dataset. It is a synthetically generated dataset representing a social network. It is being used in LDBC audited benchmarks, SNB interactive, and SNB Buissines intelligence benchmarks. Keep in mind that this is NOT an official implementation of an LDBC benchmark, the open-source dataset is being used as a basis for benchmarks, and it will be used for our in-house testing process and improving Memgraph.
-
Postgres: The Graph Database You Didn't Know You Had
I designed and maintain several graph benchmarks in the Linked Data Benchmark Council, including workloads aimed for databases [1]. We make no restrictions on implementations, they can any query language like Cypher, SQL, etc.
In our last benchmark aimed at analytical systems [2], we found that SQL queries using WITH RECURSIVE can work for expressing reachability and even weighted shortest path queries. However, formulating an efficient algorithm yields very complex SQL queries [3] and their execution requires a system with a sophisticated optimizer such as Umbra developed at TU Munich [4]. Industry SQL systems are not yet at this level but they may attain that sometime in the future.
Another direction to include graph queries in SQL is the upcoming SQL/PGQ (Property Graph Queries) extension. I'm involved in a project at CWI Amsterdam to incorporate this language into DuckDB [5].
[1] https://ldbcouncil.org/benchmarks/snb/
[2] https://www.vldb.org/pvldb/vol16/p877-szarnyas.pdf
[3] https://github.com/ldbc/ldbc_snb_bi/blob/main/umbra/queries/...
[4] https://umbra-db.com/
[5] https://www.cidrdb.org/cidr2023/slides/p66-wolde-slides.pdf
- Bullshit Graph Database Performance Benchmarks
-
From Data Preprocessing to Using Graph Database
Pull the source code from https://github.com/ldbc/ldbc_snb_datagen/tree/stable.To generate data for scale factor 1-1000, use the stable branch.
hash-db
-
CRDT-richtext: Rust implementation of Peritext and Fugue
https://github.com/samsquire/hash-db
I need to combine the ideas in each of these projects into a cohesive solution.
I did some work on trying to implement the YATA algorithm, poorly.
-
Ask HN: How do you test SQL?
From an SQL database implementation perspective, in my toy Python barebones SQL database that barely supports inner joins (https://github.com/samsquire/hash-db) I tested by testing on postgresql and seeing if my query with two joins produces the same results.
I ought to produce unit tests that prove that tuples from each join operation produces the correct dataset.
For a user perspective, I guess you could write some tooling that loads example data into a database and does an incremental join with each part of the join statement added.
-
Bullshit Graph Database Performance Benchmarks
I wrote a toy dynamodb, SQL, Cypher graph and document storage database engine in Python for the learning.
https://github.com/samsquire/hash-db
- Experimental distributed keyvalue database (it uses python dictionaries) imitating dynamodb querying with join only SQL support, distributed joins and simple Cypher graph support
-
How necessary are the programming fundamentals?
I am interested in database internals. Btrees come up with regard to designing database systems that are efficient to query on disk. Postgres uses them for its indexes. Radix trees are memory efficient tries which are useful for answering prefix queries. They're also called prefix trees. I use them to get a list of prefixes of a string. Useful for simple intellisense style forms or dynamodb style querying. I've also been studying LSM trees which are used in Leveldb and RocksDB.
I experiment with database technology in my experimental project hash-db https://github.com/samsquire/hash-db The code should be readable.
I need to change my search tree to be self balancing currently it grows to the left or right without balancing. I think I need to use tree rotation depending on which branch has the highest height.
What are some alternatives?
Apache AGE - Graph database optimized for fast analysis and real-time data processing. It is provided as an extension to PostgreSQL.
electric - Local-first sync layer for web and mobile apps. Build reactive, realtime, local-first apps directly on Postgres.
ldbc_snb_bi - Reference implementations for the LDBC Social Network Benchmark's Business Intelligence (BI) workload
kuzu - Embeddable property graph database management system built for query speed and scalability. Implements Cypher.
benchgraph
dbt-unit-testing - This dbt package contains macros to support unit testing that can be (re)used across dbt projects.
arcadedb - ArcadeDB Multi-Model Database, one DBMS that supports SQL, Cypher, Gremlin, HTTP/JSON, MongoDB and Redis. ArcadeDB is a conceptual fork of OrientDB, the first Multi-Model DBMS. ArcadeDB supports Vector Embeddings.
ustore - Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️
simple-graph - This is a simple graph database in SQLite, inspired by "SQLite as a document database"
pg_crdt - POC CRDT support in Postgres
nebula-docker-compose - Docker compose for Nebula Graph
data-diff - Compare tables within or across databases