Bullshit Graph Database Performance Benchmarks

• Memgraph

  44 2,086 9.7 C++

  Open-source graph database, tuned for dynamic analytics environments. Easy to adopt, scale and own.

  

For more context:

- blog post that sparked the discussion - https://memgraph.com/blog/memgraph-vs-neo4j-performance-benc...

- earlier discussion about this Memgraph benchmark HackerNews - https://news.ycombinator.com/item?id=33813781

- the benchmark results - https://memgraph.com/benchgraph/

- benchmark repo and methodology - https://github.com/memgraph/memgraph/tree/master/tests/mgben...

• ldbc_snb_datagen_spark

  5 162 5.0 Java

  Synthetic graph generator for the LDBC Social Network Benchmark, running on Spark

  

A plug: if you are looking for TPC-style application-level benchmarks for database systems, check out the LDBC Social Network Benchmark [1]. It has workloads for both OLTP and OLAP systems. We designed both of these to prevent many of the common benchmarking mistakes. To ensure that implementations follow the specification and their results are reproducible, we have a rigorous auditing process (similarly to TPC's benchmarks) [2].

[1] https://ldbcouncil.org/docs/presentations/ldbc-snb-2021-12.p...

[2] https://ldbcouncil.org/benchmarks/snb/

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• kuzu

  11 1,014 9.9 C++

  Embeddable property graph database management system built for query speed and scalability. Implements Cypher.

  

• simple-graph

  7 1,329 2.5 Go

  This is a simple graph database in SQLite, inspired by "SQLite as a document database"

• ustore

  15 486 9.6 C++

  Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️

  

I am really stunned by this story. It made me check the MemGraph benchmarks section. Don't get me wrong, it may be 10-100x faster than Neo4J in even the most basic operations. Moreover, given the quality of Neo4J, it is hard not to be that much quicker. Even Postgres and MySQL are better at storing graphs than Neo4J.

---

Disclosure: I have worked on Graph Algorithms, Graph Databases, and Database Engines for years, and we are now preparing a commercial solution based on UKV [1]. I don't know anyone at MemGraph or Neo4J. Never used the first. As for the second, I am not a fan.

---

Aside from licensing, there are 3 primary complaints. I will address them individually, and I am open to a discussion.

A. Using Python for Benchmarks instead of Gatling. I don't entirely agree with this. Python still has the fastest-growing programming community while already being one of the 2 most popular languages. Gatling, however, never heard of it. Choosing between the two, I would pick Python. But neither works if you want to design a High-Performance benchmark for a fast system. Without automatic memory management and expensive runtimes, you can only implement those in C, C++, Rust, or another systems-programming language. We have faced that too many times that the benchmark itself works worse than the system it is trying to evaluate [2].

B. Using hardware from 2010 [3], weird datasets [4]. This shocked me. When I looked at the charts [5] and the benchmarking section, it seemed highly professional and good-looking. I wouldn't expect less from a startup with $20M VC funding. But the devil is in the details. I would have never expected anyone benchmarking a new DBMS to use now 13-year-old CPUs and an unknown dataset. Assuming current developer salaries, hiring people to design a DBMS doesn't make sense if you will be evaluating on a $1000 machine is just financially irresponsible. We buy expensive servers, they cost like sports cars or even apartments in poorer countries. It is hard to maintain, but they are essential to quality work. It is sad to see companies taking such shortcuts. But to be a devil's advocate, there is no 1 graph benchmark or dataset that everyone agrees on. So I imagine people experimenting with multiple real datasets of different sizes or generating them systemically using one of the Random Generator algorithms. In UKV, we have used Twitter data to construct both document and graph collections. In the past, we have also used `ci-patent`, `bio-mouse-gene`, `human-Jung2015-M87102575`, and hundreds of other public datasets from the Network Repository and SNAP [6]. There are datasets of every shape and size, reaching around 1 Billion edges, in case someone is searching for data. For us the next step is the reconstruction of the Web from the 300 TB CommonCrawl dataset [7]. There is no such Graph benchmark in existence, but it is the biggest public dataset we could find.

C. Running query different number of times for various engines. This can be justified, and it is how current benchmarks are done. You are tracking not just the mean execution time but also variability, so if at some point results converge, you abrupt before hitting the expected iterations number to save time.

---

LDBC [8] seems like a good contestant for a potential industry standard, but it needs to be completed. Its "Business Intelligence workload" and "Interactive workload" categories exclude any real "Graph Analytics". Running an All-Pairs-Shortest-Paths algorithm on a large external memory graph could have been a much more interesting integrated benchmark. Similarly, one can make large-scale community detection or personalized recommendations based on Graphs and evaluate the overall cost/performance. It, however, poses another big challenge. Almost all algorithm implementations for those problems are vertex-centric. They scale poorly with large sparse graphs that demand edge-centric algorithms, so a new implementation has to be written from scratch. We will try to allocate more resources towards that in 2023 and invite anyone curious to join.

---

[1]: https://github.com/unum-cloud/ukv

• hash-db

  5 50 0.0 Python

  Experimental distributed pseudomultimodel keyvalue database (it uses python dictionaries) imitating dynamodb querying with join only SQL support, distributed joins and simple Cypher graph support and document storage

I wrote a toy dynamodb, SQL, Cypher graph and document storage database engine in Python for the learning.

https://github.com/samsquire/hash-db

• Apache AGE

  132 2,695 8.9 C

  Graph database optimized for fast analysis and real-time data processing. It is provided as an extension to PostgreSQL. (by apache)

  

I find projects like apache/age [0] are very promising in this direction. But I wouldn't call graph db's a passing fade. A more appropriate description might be "too important to be left alone, yet not important enough to form a second type of mass market database engine".

[0] https://github.com/apache/age

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• arcadedb

  4 440 9.7 JavaScript

  ArcadeDB Multi-Model Database, one DBMS that supports SQL, Cypher, Gremlin, HTTP/JSON, MongoDB and Redis. ArcadeDB is a conceptual fork of OrientDB, the first Multi-Model DBMS. ArcadeDB supports Vector Embeddings.

  

Suggest a related project