promscale VS TimescaleDB

Now that Promscale has been deprecated, what are the other ideal means of self-hosted long term Prometheus storage?

Oh wow, I browsed the project just a few weeks ago, didn't see it then. I see the deprecation is recent (https://github.com/timescale/promscale/issues/1836)

They say:

> if you want to have a seamless experience between metrics and traces, then current experience of stitching together Prometheus & Jaeger is not great.

But I wonder if using Promscale https://github.com/timescale/promscale would make Prometheus & Jaeger not such a big problem as SigNoz imply.

Promscale readme:

> Promscale is a unified metric and trace observability backend for Prometheus, Jaeger and OpenTelemetry built on PostgreSQL and TimescaleDB.

Either way, SigNoz seems interesting indeed. And am glad to see that SigNoz supports OpenTelemetry.

Hi! So the team is over 100 at this point, but engineering effort is spread across multiple products at this point.

The core timescaledb repo [0] has 10-15 primary engineers (although we are aggressively hiring for database internal engineers), with a few others working on DB hyperfunctions and our function pipelining [1] in a separate extension [2]. I think generally the set of folks who contribute to low-level database internals in C is just smaller than other type of projects.

We also have our promscale product [3], which is our observability backend powered by SQL & TimescaleDB.

And then there is Timescale Cloud, which is obviously a large engineering effort (most of which does not happen in public repos).

And we are hiring. Fully remote & global.

https://www.timescale.com/careers

[0] https://github.com/timescale/timescaledb

[1] https://www.timescale.com/blog/function-pipelines-building-f...

[2] https://github.com/timescale/timescaledb-toolkit

[3] https://github.com/timescale/promscale ; https://github.com/timescale/tobs

Promscale is a connector for Prometheus, one of the leading open-source monitoring solutions. Promscale is developed by Timescale, a time series database with full compatibility to Postgres. Since logs are time series events, Timescale developed Promscale to ingest events from Prometheus and make them available in SQL. You can install Promscale in numerous ways.

In case it helps, Promscale (from Timescale) offers long-term storage for Prometheus data and supports both PromQL and SQL queries. Here's the project page: https://www.timescale.com/promscale/ and the repo is here https://github.com/timescale/promscale It also support OpenTelemetry tracing if that's of interest.

At first, let's give the definition of `time series`. This is a series of (timestamp, value) pairs ordered by timestamp. The `value` may contain arbitrary data - a floating-point value, a text, a json, a data structure with many columns, etc. Each time series is uniquely identified by its name plus an optional set of {label="value"} labels. For example, temperature{city="London",country="UK"} or log_stream{host="foobar",datacenter="abc",app="nginx"}.

ClickHouse is perfectly optimized for storing and querying of such time series, including metrics. That's true that ClickHouse isn't optimized for handling millions of tiny inserts per second. It prefers infrequent batches with big number of rows per each batch. But this isn't the real problem in practice, because:

1) ClickHouse provides Buffer table engine for frequent inserts.

2) It is easy to create a special proxy app or library for data buffering before sending it to ClickHouse.

TimescaleDB provides Promscale [1] - a service, which allows using TimescaleDB as a storage backend for Prometheus. Unfortunately, it doesn't show outstanding performance comparing to Prometheus itself and to other remote storage solutions for Prometheus. Promscale requires more disk space, disk IO, CPU and RAM according to production tests [2], [3].

[1] https://github.com/timescale/promscale

[2] https://abiosgaming.com/press/high-cardinality-aggregations/

[3] https://valyala.medium.com/promscale-vs-victoriametrics-reso...

Full disclosure: I'm CTO at VictoriaMetrics - competing solution for TimescaleDB. VictoriaMetrics is built on top of architecture ideas from ClickHouse.

Promscale + TimescaleDB

timescaledb A solution by Timescale. Provides a lot more functions to handle time series than pg_timeseries. Low latency makes it adequate for real-time analytics. Supports incremental views through continuous aggregates. Has some overlap with pg_mooncake, but can't write to Iceberg or Delta Lake, using them directly as the storage layer. Supports tiered storage

Even for timeseries there is https://github.com/timescale/timescaledb. Haven't used it, just knew it existed.

In fact, even in the absence of these commercial databases, users can effortlessly install PostgreSQL and leverage its built-in pgvector functionality for vector search. PostgreSQL stands as the benchmark in the realm of open-source databases, offering comprehensive support across various domains of database management. It excels in transaction processing (e.g., CockroachDB), online analytics (e.g., DuckDB), stream processing (e.g., RisingWave), time series analysis (e.g., Timescale), spatial analysis (e.g., PostGIS), and more. For non-professional users seeking to explore vector databases, they can readily download the open-source PostgreSQL or utilize managed services like Supabase and Neon to establish their own basic AI applications. Other than PostgreSQL, several open-source databases, including OpenSearch, ClickHouse, and Cassandra, have implemented their own vector search functionality. You do not need to adopt a new vector database if you have already used these systems.

Extensions for Days: PostGIS for geospatial wizardry, TimescaleDB for time-series data, pg_trgm for fuzzy search. It’s like Skyrim modded do whatever you want.

In this post, we’ll explore practical SQL patterns for retrieving the latest sensor readings efficiently, and how to use continuous aggregates for historical analytics—based on real-world deployments using TimescaleDB on PostgreSQL.

I had to investigate a bit into TimescaleDB as well on which the analytics events are stored since it allows for efficient time series queries.

In Part 2 of our in-depth Replicas tutorial series, we dive into the implementation and monitoring of HA (High Availability) Replicas and Read Replicas in TimescaleDB.

for one thing, depending on your licensing contraints:

- Citus is AGPLv3 https://github.com/citusdata/citus/blob/v13.0.1/LICENSE

- Hydra is Apache 2 https://github.com/hydradatabase/columnar/blob/v1.1.2/LICENS...

- Timescale is mostly Apache 2 https://github.com/timescale/timescaledb/blob/2.18.2/LICENSE

This is where TimescaleDB comes in. Built on PostgreSQL (it’s an extension), TimescaleDB is purpose-built for time series and other demanding workloads, and thanks to the timescaledb gem, it integrates seamlessly into Rails. You don’t have to leave behind the conventions or patterns you love, it just works alongside them.

For those of you who don’t know, TimescaleDB is a PostgreSQL extension for high-performance real-time analytics on time series and event data. It is available as an open-source extension or fully managed on Timescale Cloud.