timescale-analytics VS orioledb

Hi! So the team is over 100 at this point, but engineering effort is spread across multiple products at this point.

The core timescaledb repo [0] has 10-15 primary engineers (although we are aggressively hiring for database internal engineers), with a few others working on DB hyperfunctions and our function pipelining [1] in a separate extension [2]. I think generally the set of folks who contribute to low-level database internals in C is just smaller than other type of projects.

We also have our promscale product [3], which is our observability backend powered by SQL & TimescaleDB.

And then there is Timescale Cloud, which is obviously a large engineering effort (most of which does not happen in public repos).

And we are hiring. Fully remote & global.

https://www.timescale.com/careers

[0] https://github.com/timescale/timescaledb

[1] https://www.timescale.com/blog/function-pipelines-building-f...

[2] https://github.com/timescale/timescaledb-toolkit

[3] https://github.com/timescale/promscale ; https://github.com/timescale/tobs

(NB: Post author here)

This is in the TimescaleDB Toolkit extension [1] which is licensed under our community license for now and it's not available on DO. It is available on our cloud service fully managed. You can also install it and run it for free yourself.

[1]: https://github.com/timescale/timescaledb-toolkit

Absolutely! We're actually developing a lot of that: https://github.com/timescale/timescaledb-toolkit/tree/main/d...

A number of the things you're looking for we've done experimentally and we'll be stabilizing over the next few releases. So we'd love some feedback while we're still able to futz with the API without making breaking changes.

But the two you're asking about are, I think, going to be covered by hyperloglog (we just reimplemented the internals with HLL++) and stats_agg family of functions, which have both 1D (which will give you avg, stddev, variance, etc) and 2D (co-variance, slope, intercept, x-intercept etc as well as all the 1D functions).

Would also love issues if you think we're missing other stuff, going to be generalizing this and want to make it useful for folks.

(NB: Post author here.)

If you know that you're going to be doing downsampling at the hourly level then a continuous aggregate on the hour is probably a good idea. We're also building some functions to make some of the continuous aggregate stuff for these sorts of cases easier/more accurate in more cases, especially if you need things like exact averages when you don't have the same number of points in an hour and want to re-aggregate on top of the continuous agg. See: https://github.com/timescale/timescale-analytics/pull/141/files

Fair point about adaptive chunking. You sound like a long-term user!

There is always a trade-off between getting features to users quickly to experiment and incrementally improve, versus doing it always very conservatively.

When we launched adaptive chunking (introduced in 0.11, deprecated in 1.2), we explicitly marked it as beta and default off, to hopefully reflect that. [1]

The approach we are now taking with Timescale Analytics [2] is to have an explicit distinction between experimental features (which will be part of a distinct"experimental" schema in the database, and must be expressly turned on with appropriate warnings) and stable features. Hopefully this can help find a good balance between stability and velocity, but feedback welcome!

[1] https://github.com/timescale/timescaledb/releases/tag/0.11.0

[2] https://github.com/timescale/timescale-analytics/tree/main/e...

hey hn, supabase ceo here

we've been fans of Oriole for a while now and have been long-time supporters

in case you're jumping straight to the comments: OrioleDB is a table storage extension for Postgres. it acts as a drop-in replacement for the default postgres storage engine using the Table Access Method APIs (pluggable storage). the storage engine changes the representation of table data on disk. its architecture is designed to take advantage of modern hardware like SSDs and NVRAM. it implements MVCC, the feature that allows allows multiple connected users to see different versions of the data depending on when their transaction started, via an UNDO log rather than tuple versioning.

one caveat: it requires several patches to the postgres core to expand on the type of features external storage engines extensions can implement. for this reason it could be a while before you see this land as a default engine on supabase. we will probably make it available as an option for customers who want to experiment - no timeline is decided yet.

finally, we have been working with the team on decoupled storage and compute [0]. this is experimental but promising, especially with some recent advances in S3 (specifically Express One Zone [1]). we have a demonstration in the blog post.

i'll message Alexander in case there are any technical questions

[0] https://github.com/orioledb/orioledb/blob/main/doc/usage.md#...

[1] https://aws.amazon.com/s3/storage-classes/express-one-zone/

When I saw "cloud native" I was expecting S3-ish the way Neon does it but they say it's experimental: https://github.com/orioledb/orioledb/blob/beta4/doc/usage.md... and for them to say "beta, don't use in production" and then a separate "experimental" label must make it really bad

There are some interesting things in development to potentially solve that problem.

Here's a recent HN submission about OrioleDB of the more promising ones: https://news.ycombinator.com/item?id=36740921

Source code: https://github.com/orioledb/orioledb

https://github.com/orioledb/orioledb/blob/main/doc/arch.md

> - PostgreSQL is very conservative (maybe extremely) conservative about data safety (mostly achieved via fsync-ing at the right times), and that propagates through the IO stack, including SSD firmware, to cause slowdowns

This is why our first goal is to become pure extension. Becoming part of PostgreSQL would require test of time.

> - MVCC is very nice for concurrent access - the Oriole doc doesn't say with what concurrency are the graphs achieved

Good catch. I've added information about VM type and concurrency to the blog post.

> - The title of the Oriole doc and its intro text center about solving VACUUM, which is of course a good goal, but I don't think they show that the "square wave" graphs they achieve for PostgreSQL are really in majority caused by VACUUM. Other benchmarks, like Percona's (https://www.percona.com/blog/evaluating-checkpointing-in-pos...) don't yield this very distinctive square wave pattern.

Yes, it's true. The square patters is because of checkpointing. The reason of improvements here is actually not VACUUM, but modification of relevant indexes only (and row-level WAL, which decreases overall IO).

I took a look at https://github.com/orioledb/orioledb which is a project attempting to remedy some of Postgres' shortcomings, including MVCC. It looks like they're doing something similar to MySQL with a redo log, as well as some other optimizations. So maybe this is the answer.

You don’t need a database written (or rewritten in Rust). we’re working to make Postgres scalable for the next decade too https://github.com/orioledb/orioledb

> I’d love to see B-Tree primary storage option. Aka store the row data inside the primary index.

It is coming: https://github.com/orioledb/orioledb

sorry to underwhelm!

if you like Neon, then I imagine you like their database branching model? On Friday we announced[0] our 500K investment into OrioleDB, who are working on branching[1], with the plan to upstream these changes into Postgres core.

It would be possible for us to run a fork of Postgres today which supports branching, but our long-term view is that developers would prefer a non-forked version of Postgre (to mitigate any risk of lock-in). So we will work on adding branching to Postgres core in the background, which will be a benefit to the entire Postgres ecosystem.

[0] Announcement:https://supabase.com/blog/supabase-series-b#where-were-going

[1] https://github.com/orioledb/orioledb/wiki/Database-branching