denorm VS timescale-analytics

> Continuous aggregation

I found this to be a very common need, so I created [denorm](https://github.com/rivethealth/denorm) for doing this in vanilla PostgreSQL. (It also does incrementally updated joins as well.)

(I expect the performance of a native implementation to be superior. Unfortunately, I use managed database services.)

Hi! So the team is over 100 at this point, but engineering effort is spread across multiple products at this point.

The core timescaledb repo [0] has 10-15 primary engineers (although we are aggressively hiring for database internal engineers), with a few others working on DB hyperfunctions and our function pipelining [1] in a separate extension [2]. I think generally the set of folks who contribute to low-level database internals in C is just smaller than other type of projects.

We also have our promscale product [3], which is our observability backend powered by SQL & TimescaleDB.

And then there is Timescale Cloud, which is obviously a large engineering effort (most of which does not happen in public repos).

And we are hiring. Fully remote & global.

https://www.timescale.com/careers

[0] https://github.com/timescale/timescaledb

[1] https://www.timescale.com/blog/function-pipelines-building-f...

[2] https://github.com/timescale/timescaledb-toolkit

[3] https://github.com/timescale/promscale ; https://github.com/timescale/tobs

(NB: Post author here)

This is in the TimescaleDB Toolkit extension [1] which is licensed under our community license for now and it's not available on DO. It is available on our cloud service fully managed. You can also install it and run it for free yourself.

[1]: https://github.com/timescale/timescaledb-toolkit

Absolutely! We're actually developing a lot of that: https://github.com/timescale/timescaledb-toolkit/tree/main/d...

A number of the things you're looking for we've done experimentally and we'll be stabilizing over the next few releases. So we'd love some feedback while we're still able to futz with the API without making breaking changes.

But the two you're asking about are, I think, going to be covered by hyperloglog (we just reimplemented the internals with HLL++) and stats_agg family of functions, which have both 1D (which will give you avg, stddev, variance, etc) and 2D (co-variance, slope, intercept, x-intercept etc as well as all the 1D functions).

Would also love issues if you think we're missing other stuff, going to be generalizing this and want to make it useful for folks.

(NB: Post author here.)

If you know that you're going to be doing downsampling at the hourly level then a continuous aggregate on the hour is probably a good idea. We're also building some functions to make some of the continuous aggregate stuff for these sorts of cases easier/more accurate in more cases, especially if you need things like exact averages when you don't have the same number of points in an hour and want to re-aggregate on top of the continuous agg. See: https://github.com/timescale/timescale-analytics/pull/141/files

Fair point about adaptive chunking. You sound like a long-term user!

There is always a trade-off between getting features to users quickly to experiment and incrementally improve, versus doing it always very conservatively.

When we launched adaptive chunking (introduced in 0.11, deprecated in 1.2), we explicitly marked it as beta and default off, to hopefully reflect that. [1]

The approach we are now taking with Timescale Analytics [2] is to have an explicit distinction between experimental features (which will be part of a distinct"experimental" schema in the database, and must be expressly turned on with appropriate warnings) and stable features. Hopefully this can help find a good balance between stability and velocity, but feedback welcome!

[1] https://github.com/timescale/timescaledb/releases/tag/0.11.0

[2] https://github.com/timescale/timescale-analytics/tree/main/e...