ldbc_snb_bi
spicedb
ldbc_snb_bi | spicedb | |
---|---|---|
3 | 38 | |
33 | 4,595 | |
- | 4.0% | |
7.7 | 9.7 | |
4 months ago | 4 days ago | |
Python | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ldbc_snb_bi
-
Demand the impossible: rigorous database benchmarking
Rigorous database benchmarking is indeed very difficult and time-consuming. I spent the last ~7 years working on benchmarks for graph processing systems in the Linked Data Benchmark Council (LDBC) [1], originally established in 2012 as an EU research project.
LDBC creates TPC-style application-level database benchmarks which can be used for system-to-system comparison. We provide detailed specifications, data generators, benchmark frameworks, and multiple reference implementations. The benchmarks are implemented by vendors for their database products, and the implementations submitted to be run by independent third-party auditors to ensure their correctness and reproducibility.
We have found that there is a market for audits for graph processing systems, albeit it is quite small: over the last 4 years, we have published 34 audited results, see e.g. [2] and [3].
A major problem we face is that process of implementing the benchmark for a system and getting an audited result is long (and therefore expensive). Vendors spend months implementing the and tuning the benchmarks. It is also typical for the auditor to spend 50+ hours on the auditing process, which includes a lengthy code review step, setting up the system, running the experiments, testing ACID properties, writing a report, etc. The length of the process is exacerbated by the lack of standard graph query languages. This potentially necessitates the auditor to learn a new query language or programming language.
We have tried to mitigate this problem by improving our documentation, creating more reference implementation, distributing pre-generated data sets. There are new standard graph query languages (SQL/PGQ, GQL) but their adoption is still very limited. Overall, the auditing process is quite long, which is mainly caused by the essential complexity of the problem: implementing an application-level benchmark and getting reliable results is very difficult.
[1] https://ldbcouncil.org/introduction/
[2] https://ldbcouncil.org/benchmarks/snb-interactive
[3] https://ldbcouncil.org/benchmarks/snb-bi/
-
Benchgraph Backstory: The Untapped Potential
At first, the plan was to use only the LDBC dataset and write different queries for the dataset, but LDBC has a set of well-designed queries that were specifically prepared to stress the database. Each query targets a special scenario, also called “chock point.” Not to be mistaken, they do not have deep graph traversal doing around 100 hops, but they are definitely more complex than the ones written for the Pokec dataset. There are two sets of queries for the LDBC SNB: interactive and business intelligence. LDBC provides a reference Cypher implementation for both of these queries for Neo4j. We took those queries, tweaked the data types, and made the queries work on Memgraph. Again, to be perfectly clear, this is NOT an official implementation of an LDBC Benchmark; this goes for both interactive and business intelligence queries. The queries were used as the basis for running the benchmark.
-
Postgres: The Graph Database You Didn't Know You Had
I designed and maintain several graph benchmarks in the Linked Data Benchmark Council, including workloads aimed for databases [1]. We make no restrictions on implementations, they can any query language like Cypher, SQL, etc.
In our last benchmark aimed at analytical systems [2], we found that SQL queries using WITH RECURSIVE can work for expressing reachability and even weighted shortest path queries. However, formulating an efficient algorithm yields very complex SQL queries [3] and their execution requires a system with a sophisticated optimizer such as Umbra developed at TU Munich [4]. Industry SQL systems are not yet at this level but they may attain that sometime in the future.
Another direction to include graph queries in SQL is the upcoming SQL/PGQ (Property Graph Queries) extension. I'm involved in a project at CWI Amsterdam to incorporate this language into DuckDB [5].
[1] https://ldbcouncil.org/benchmarks/snb/
[2] https://www.vldb.org/pvldb/vol16/p877-szarnyas.pdf
[3] https://github.com/ldbc/ldbc_snb_bi/blob/main/umbra/queries/...
[4] https://umbra-db.com/
[5] https://www.cidrdb.org/cidr2023/slides/p66-wolde-slides.pdf
spicedb
-
How do you manage transactions in Go? Do we really need to use one transaction for each request?
Have you taken a look at SpiceDB? The Authzed blog has a few posts that are useful to improving your understanding -- I can think of two: New Enemies and Writing relationships to SpiceDB.
-
How to start a Go project in 2023
Things I can't live without in a new Go project in no particular order:
- https://github.com/golangci/golangci-lint - meta-linter
- https://goreleaser.com - automate release workflows
- https://magefile.org - build tool that can version your tools
- https://github.com/ory/dockertest/v3 - run containers for e2e testing
- https://github.com/ecordell/optgen - generate functional options
- https://golang.org/x/tools/cmd/stringer - generate String()
- https://mvdan.cc/gofumpt - stricter gofmt
- https://github.com/stretchr/testify - test assertion library
- https://github.com/rs/zerolog - logging
- https://github.com/spf13/cobra - CLI framework
FWIW, I just lifted all the tools we use for https://github.com/authzed/spicedb
We've also written some custom linters that might be useful for other folks: https://github.com/authzed/spicedb/tree/main/tools/analyzers
-
Feature flags and authorization abstract the same concept
At AuthZed, we think about this topic regularly while developing SpiceDB[0], except we believe feature flags are a subset of authorization. I'd disagree with the author that permissions are always long-lived -- authorization can also be ephemeral (and often that's how it's most secure) or dependent on run-time context[1]. What's more, using SpiceDB, we can often collapse checking for authorization and feature-flags into a single round-trip by defining a permission that can additionally require a feature flag (e.g. permission = admin & has_feature_flag).
It's a little silly, but lots of folks ask for the moon when it comes to performance for authorization because it's critical to every request, but then go on and sprinkle a dozen feature flag RPCs each adding more and more latency. We think you should be able to have both.
What we're excited about is use cases beyond feature flags and authorization: we've also seen some folks use SpiceDB as an update graph or others as a dependency graph.
[0]: https://github.com/authzed/spicedb
[1]: https://authzed.com/blog/caveats/
-
Postgres: The Graph Database You Didn't Know You Had
It scaled well compared to a naive graph abstraction implemented outside the database, but when performance wasn't great, it REALLY wasn't great. We ended up throwing it out in later versions to try and get more consistent performance.
I've since worked on SpiceDB[1] which takes the traditional design approach for graph databases and simply treating Postgres as triple-store and that scales far better. IME, if you need a graph, you probably want to use a database optimized for graph access patterns. Most general-purpose graph databases are just bags of optimizations for common traversals.
[0]: https://github.com/quay/clair
[1]: https://github.com/authzed/spicedb
-
Writing a Kubernetes Operator
I get the sentiment. We held off on building an operator until we felt there was actually value in doing so (for the most part, Deployments cover the operational needs pretty well).
Migrations can be run in containers (and they are, even with the operator), but it's actually a lot of work to run them at the right time, only once, with the right flags, in the right order, waiting for SpiceDB to reach a specific spot in a phased migrations, etc.
Moving from v1.13.0 to v1.14.0 of SpiceDB requires a multi-phase migration to avoid downtime[0], as could any phased migration for any stateful workload. The operator will walk you through them correctly, without intervention. Users who aren't running on Kubernetes or aren't using the operator often have problems running these steps correctly.
The value is in this automation, but also in the API interface itself. RDS is just some automation and an API on top of EC2, and I think RDS has value over running postgres on EC2 myself directly.
As for helm charts, this is just my opinion, but I don't think they're a good way to distribute software to end users. The interface for a helm chart becomes polluted over time in the same way that most operator APIs become polluted over time, as more and more configuration is pulled up to the top. I think helm is better suited to managing configuration you write yourself to deploy on your own clusters (I realize I'm in the minority here).
[0]: https://github.com/authzed/spicedb/releases/tag/v1.14.0
- AWS Creates New Policy-Based Access Control Language Cedar
-
Solution for ReBAC authz using attributes?
To my understanding, the only ReBAC system that supports dynamic attributes is SpiceDB.
-
The Annotated Google Zanzibar Paper
If you're curious to see a Postgres-based implementation, SpiceDB has a Postgres driver: https://github.com/authzed/spicedb/tree/main/internal/datast...
- We built an open source authorization service based on Google Zanzibar
-
One Million Database Connections
Interesting, for SpiceDB[0], one place we've struggled with MySQL is preemptively establishing connections in the pool so that it's always full. PGX[1] has been fantastic for Postgres and CockroachDB, but I haven't found something with enough control for MySQL.
[0]: https://github.com/authzed/spicedb
What are some alternatives?
ldbc_snb_datagen_spark - Synthetic graph generator for the LDBC Social Network Benchmark, running on Spark
Ory Keto - Open Source (Go) implementation of "Zanzibar: Google's Consistent, Global Authorization System". Ships gRPC, REST APIs, newSQL, and an easy and granular permission language. Supports ACL, RBAC, and other access models.
ldbc_snb_interactive_v1_impls - Reference implementations for LDBC Social Network Benchmark's Interactive workload.
OPA (Open Policy Agent) - Open Policy Agent (OPA) is an open source, general-purpose policy engine.
materialize - The data warehouse for operational workloads.
casbin - An authorization library that supports access control models like ACL, RBAC, ABAC in Golang: https://discord.gg/S5UjpzGZjN
Apache AGE - Graph database optimized for fast analysis and real-time data processing. It is provided as an extension to PostgreSQL.
realworld - "The mother of all demo apps" — Exemplary fullstack Medium.com clone powered by React, Angular, Node, Django, and many more
clair - Vulnerability Static Analysis for Containers
zanzibar-pg - Pure PL/pgSQL implemenation of the Zanzibar API
quine - Quine • a streaming graph • https://quine.io • Discord: https://discord.gg/GMhd8TE4MR
oso - Oso is a batteries-included framework for building authorization in your application.