noisepage VS orioledb

> Carne

Okay, so, noisepage appears to be open source https://github.com/cmu-db/noisepage/

But I can't find the Ottertune Github page

Is any part of Ottertune open source?

I was one of the main authors of a research project called Peloton (https://github.com/cmu-db/peloton) which was later rebranded to NoisePage (https://github.com/cmu-db/noisepage). The initial version of RisingWave actually borrowed a lot from Peloton (fun fact: that's also how DuckDB https://duckdb.org/ started!), but we decided to rewrite in Rust due to development cost and security (e.g., memory leakage) considerations (more info: https://www.risingwave-labs.com/blog/building-a-cloud-database-from-scratch-why-we-moved-from-cpp-to-rust/).

> If I may, can you please shed light on why Peloton had to be archived and in essence re-done with OtterTune. Interested in your team's learnings from it from a software engineering point of view.

Peloton and OtterTune are completely different projects. Peloton was abandoned and rewritten as NoisePage (https://noise.page). OtterTune has always been OtterTune.

See this recent interview where I discuss why we gave up on Peloton:

https://www.ibm.com/cloud/blog/database-deep-dives-with-andy...

> - How did the team ensure this project doesn't suffer from the same disadvantages as its predecessor?

Again, different projects. OtterTune is all about not having to modify the internals of Postgres, MySQL, and any other DBMS. This is why we were able to support Oracle in the academic version in a short amount of time:

https://ottertune.com/blog/vldb-autonomous-database-tuning-i...

> - What would you advise other teams undertaking a rewrite to pay off their tech debts?

It is hard for to provide general advice for this question because every situation is different.

> How does this project compare to / contrast with Google's and SingleStore's efforts in this space?

I am not familiar with Google or SingleStore using ML in the manner that we are with OtterTune to tune configuration knobs. Or at least I have not seen anything public about it.

These days Oracle is the most aggressive with pushing automated tuning capabilities (Oracle's autonomous DBaaS, AutoPilot for MySQL Heatwave). The difference with these approaches and OtterTune is that right now we are focused on configuration tuning (to avoid data privacy issues) and our core approach is platform/DBMS agnostic.

> Any chance we see you do a Peter Bailis and Sisu Data this? (:

I don't know what you mean by this? Peter Bailis is the Ryan Gosling of databases.

The state of the art for this sort of thing is (Leanstore/Umbra - https://umbra-db.com/) or the new NoisePage database (https://github.com/cmu-db/noisepage/tree/master/src/storage). There is also the HyRise database, but that one focuses more on datasets that fit entirely in memory (https://hpi.de/plattner/projects/hyrise.html)

hey hn, supabase ceo here

we've been fans of Oriole for a while now and have been long-time supporters

in case you're jumping straight to the comments: OrioleDB is a table storage extension for Postgres. it acts as a drop-in replacement for the default postgres storage engine using the Table Access Method APIs (pluggable storage). the storage engine changes the representation of table data on disk. its architecture is designed to take advantage of modern hardware like SSDs and NVRAM. it implements MVCC, the feature that allows allows multiple connected users to see different versions of the data depending on when their transaction started, via an UNDO log rather than tuple versioning.

one caveat: it requires several patches to the postgres core to expand on the type of features external storage engines extensions can implement. for this reason it could be a while before you see this land as a default engine on supabase. we will probably make it available as an option for customers who want to experiment - no timeline is decided yet.

finally, we have been working with the team on decoupled storage and compute [0]. this is experimental but promising, especially with some recent advances in S3 (specifically Express One Zone [1]). we have a demonstration in the blog post.

i'll message Alexander in case there are any technical questions

[0] https://github.com/orioledb/orioledb/blob/main/doc/usage.md#...

[1] https://aws.amazon.com/s3/storage-classes/express-one-zone/

When I saw "cloud native" I was expecting S3-ish the way Neon does it but they say it's experimental: https://github.com/orioledb/orioledb/blob/beta4/doc/usage.md... and for them to say "beta, don't use in production" and then a separate "experimental" label must make it really bad

There are some interesting things in development to potentially solve that problem.

Here's a recent HN submission about OrioleDB of the more promising ones: https://news.ycombinator.com/item?id=36740921

Source code: https://github.com/orioledb/orioledb

https://github.com/orioledb/orioledb/blob/main/doc/arch.md

> - PostgreSQL is very conservative (maybe extremely) conservative about data safety (mostly achieved via fsync-ing at the right times), and that propagates through the IO stack, including SSD firmware, to cause slowdowns

This is why our first goal is to become pure extension. Becoming part of PostgreSQL would require test of time.

> - MVCC is very nice for concurrent access - the Oriole doc doesn't say with what concurrency are the graphs achieved

Good catch. I've added information about VM type and concurrency to the blog post.

> - The title of the Oriole doc and its intro text center about solving VACUUM, which is of course a good goal, but I don't think they show that the "square wave" graphs they achieve for PostgreSQL are really in majority caused by VACUUM. Other benchmarks, like Percona's (https://www.percona.com/blog/evaluating-checkpointing-in-pos...) don't yield this very distinctive square wave pattern.

Yes, it's true. The square patters is because of checkpointing. The reason of improvements here is actually not VACUUM, but modification of relevant indexes only (and row-level WAL, which decreases overall IO).

I took a look at https://github.com/orioledb/orioledb which is a project attempting to remedy some of Postgres' shortcomings, including MVCC. It looks like they're doing something similar to MySQL with a redo log, as well as some other optimizations. So maybe this is the answer.

You don’t need a database written (or rewritten in Rust). we’re working to make Postgres scalable for the next decade too https://github.com/orioledb/orioledb

> I’d love to see B-Tree primary storage option. Aka store the row data inside the primary index.

It is coming: https://github.com/orioledb/orioledb

sorry to underwhelm!

if you like Neon, then I imagine you like their database branching model? On Friday we announced[0] our 500K investment into OrioleDB, who are working on branching[1], with the plan to upstream these changes into Postgres core.

It would be possible for us to run a fork of Postgres today which supports branching, but our long-term view is that developers would prefer a non-forked version of Postgre (to mitigate any risk of lock-in). So we will work on adding branching to Postgres core in the background, which will be a benefit to the entire Postgres ecosystem.

[0] Announcement:https://supabase.com/blog/supabase-series-b#where-were-going

[1] https://github.com/orioledb/orioledb/wiki/Database-branching