Dragonflydb – A modern replacement for Redis and Memcached

• dragonfly

  49 23,944 9.9 C++

  A modern replacement for Redis and Memcached

  

I want to take a minute to appreciate and recognize the https://github.com/dragonflydb/dragonfly#background section.

A lot of projects say "faster" without giving some hint of the things they did to achieve this. "A novel fork-less snapshotting algorithm", "each thread would manage its own slice of dictionary data", and "core hashtable structure" are all important information that other projects often leave out.

• Redis

  320 64,893 9.7 C

  Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.

  

Yes, the core folks in Redis now have outlined a plan for moving towards multi-threaded awhile ago. It's honestly not made a lot of progress because raw performance matters a lot less than is argued here. As was succinctly mentioned by antirez already, Redis scales comfortably to 100s of millions of QPS with cluster mode. So, it's really building a lot of custom functionality to support better vertical scaling. Which is useful, especially when the vertical scaling keeps you on a single process.

The conversation happened here, https://github.com/redis/redis/issues/8340, and it's not like the most pressing issue for the project. It's also not as complex as what was implemented for dragonfly, which basically has native support from the ground up for concurrent programming during command execution. It would be hard to do in C as well.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• helio

  1 398 9.0 C++

  A modern framework for backend development based on io_uring Linux interface

I like the redis protocol compatibility and the HTTP compatibility, but from the initial skim through I guess you are using abseil-cpp and the home-grown helio (https://github.com/romange/helio) library.

Could you get me a one liner on the helio library is it used as a fiber wrapper around the io_uring facility in the kernel? Can it be used as a standalone library for implementing fibers in application code?

Also it seems that spinlock has become a defacto standard in the DB world today, thanks for not falling into the trap (because 90% of the users of any DB do not need spinlocks).

Another curious question would why not implement with seastar (since you're not speaking to disk often enough)?

• midi-redis

  1 27 0.0 C++

  A toy memory store with great performance

Yes, helio is the library that allows you to build c++ backends easily similar to Seastar. Unlike Seastar that is designed as futures and continuations library, helio uses fibers which I think simpler to use and reason about. I've wrote a few blog posts a while ago about fibers and Seastar: https://www.romange.com/2018/07/12/seastar-asynchronous-c-fr... one of them. You will see there a typical Seastar flow with continuations. I just do not like this style and I think C++ is not a good fit for it. Having said that, I do think Seastar is 5-star framework and the team behind it are all superstars. I learned about shared-nothing architecture from Seastar.

Re helio: You will find examples folder inside the projects with sample backends: echo_server and pingpong_server. Both are similar but the latter speaks RESP. I also implemented a toy midi-redis project https://github.com/romange/midi-redis which is also based on helio.

In fact dragonfly evolved from it.

• Aerospike

  16 972 8.7 C

  Aerospike Database Server – flash-optimized, in-memory, nosql database

  

Interesting project. Very similar to KeyDB [1] which also developed a multi-threaded scale-up approach to Redis. It's since been acquired by Snapchat. There's also Aerospike [2] which has developed a lot around low-latency performance.

1. https://docs.keydb.dev/

2. https://aerospike.com/

• rust

  2,683 93,266 10.0 Rust

  Empowering everyone to build reliable and efficient software.

  

• Apache Arrow

  75 13,562 10.0 C++

  Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

  

I've used Apache Arrow before[1]; in-memory columnar storage. We did some AI/ML stuff with data gathered from social network APIs, but you can probably do a ton of things.

[1] https://arrow.apache.org/

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• glommio

  29 2,851 7.6 Rust

  Glommio is a thread-per-core crate that makes writing highly parallel asynchronous applications in a thread-per-core architecture easier for rustaceans.

  

• deno

  448 92,975 9.9 Rust

  A modern runtime for JavaScript and TypeScript.

  

> Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.

No, there are plenty that still use permissive licenses.

GitLab uses MIT and a custom license for EE: https://docs.gitlab.com/ee/development/licensing.html

Deno uses an MIT license and has some secret sauce that is currently just in hosted services AFAIK: https://github.com/denoland/deno/blob/main/LICENSE.md

PlanetScale has hosted services and an open source tool called Vitess which is Apache licensed: https://planetscale.com/ https://github.com/vitessio/vitess

Finally Redis has a BSD licensed core, a source available license for additional modules, and a closed source license for enterprise. https://redis.com/legal/licenses/

• vitess

  60 17,861 9.9 Go

  Vitess is a database clustering system for horizontal scaling of MySQL.

  

> Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.

No, there are plenty that still use permissive licenses.

GitLab uses MIT and a custom license for EE: https://docs.gitlab.com/ee/development/licensing.html

Deno uses an MIT license and has some secret sauce that is currently just in hosted services AFAIK: https://github.com/denoland/deno/blob/main/LICENSE.md

PlanetScale has hosted services and an open source tool called Vitess which is Apache licensed: https://planetscale.com/ https://github.com/vitessio/vitess

Finally Redis has a BSD licensed core, a source available license for additional modules, and a closed source license for enterprise. https://redis.com/legal/licenses/

• amzn-drivers

  4 441 9.1 C

  Official AWS drivers repository for Elastic Network Adapter (ENA) and Elastic Fabric Adapter (EFA)

  

Of course, there are.

I was mostly running on AWS. In terms of hardware, for small packets loadtests most systems are constrained on throughput, i.e. number of packets per second. Some systems saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond some limit. Best systems networkwise are c6gn family types. They are also better than other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor.

In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem:

• webdis

  3 2,782 6.3 C

  A Redis HTTP interface with JSON output

I think the HTTP connection should be used to integrate webdis. https://github.com/nicolasff/webdis

• cachegrand

  24 963 8.0 C

  cachegrand - a modern data ingestion, processing and serving platform built for today's hardware

https://github.com/danielealbano/cachegrand :)

I am not going to get into what's better and what's not, especially because I haven't released the v0.1 yet and therefore it's not usable, but I am working on cachegrand which is aims to be (also) a redis compatible platform.

I have done A LOT of research and development before picking up the current architecture (you can see it from the amounts of commits) and I am trying to test as much as possible (Almost 1000 unit tests so far, but there is still plenty to do).

if you look at the repository please bare in mind that:

• neon

  124 12,327 9.9 Rust

  Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, branching, and bottomless storage.

  

Will be very interesting to see how it plays out. I'm yet to see the database technology completely licensed as BSL to be successful in Open Source strategy.

Elastic made change after it was very popular, MariaDB is same story, and even more so only uses BSL for "Enterprise" components which have very little community adoption.

We see however other folks, such as Neon picking permissive license for their technology https://github.com/neondatabase/neon

I think for Open Source Project just starting up concern of "Clouds will steal my lunch" is just stupid. If you're worth for clouds to Adopt you're in 0.1% of all Open Source Projects and already "winning" You can WHEN revisit your license, think how to get to that point, rather than create adoption barriers early on

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project