cortex
TimescaleDB
Our great sponsors
cortex | TimescaleDB | |
---|---|---|
17 | 82 | |
5,328 | 16,294 | |
0.8% | 1.5% | |
9.4 | 9.8 | |
6 days ago | 7 days ago | |
Go | C | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cortex
-
Self hosted log paraer
now if its more metric data you are using and want to do APM, prometheus is your man https://prometheus.io/, want to make prometheus your full time job? deploy cortex https://cortexmetrics.io/, honorable mention in the metrics space, Zabbix, https://www.zabbix.com/ I've seen use cases of zabbix going way beyond its intended use its a fantastic tool
-
Is anyone frustrated with anything about Prometheus?
Yes, but also no. The Prometheus ecosystem already has two FOSS time-series databases that are complementary to Prometheus itself. Thanos and Mimir. Not to mention M3db, developed at Uber, and Cortex, then ancestor of Mimir. There's a bunch of others I won't mention as it would take too long.
-
Centralized solution for Prometheus?
You can use the Remote write feature to send to a centralized location. It would have to be scalable like Cortex https://cortexmetrics.io/
-
Where to store high-cardinality metrics?
Cortex is not really good for high-cardinality metrics (if you are talking about https://github.com/cortexproject/cortex)
-
Building a distributed lab with an observability stack
For a homelab I think prometheus + grafana is easy to get started and scales well. There are lots of ways to set up the architecture. Prometheus can write to a directory on a filesystem, it can be set to write to a remote server, and there are other projects to integrate object storage (s3, minio, etc) or influxdb for long term storage and downsampling.
-
Prometheus federation or Thanos?
Cortex (it is renamed to Mimir recently).
- Building my first Monitoring stack - Security concerns
-
Grafana Mimir – 1B active series TSDB
Disclosure: I work for AWS, but I don't work on the Amazon Managed Service for Prometheus. I have my own very long held opinions about Free and Open Source software, and I am only speaking for myself.
To me, the AGPLv3 license isn't about forcing software users to "give changes back" to a project. It is about giving the permissions to users of software that are necessary for Software Freedom [1] when they access a program over a network. In practice, that means that changes often flow "upstream" to copyleft licensed programs one way or another. But it was never about obligating changes to be "given back" to upstream. In my personal opinion, you should be "free to fork" Free and Open Source Software (FOSS). Indeed, the Grafana folks seem to have decided to do that with Grafana Mimir.
Personally, I hope that they accept contributions under the AGPLv3 license, and hold themselves to the same obligations that others are held to with regard to providing corresponding source code of derivative works when it is made available to users over a network. In my personal opinion, too often companies use a contributor agreement that excuses them from those obligations, and also allows them to sell the software to others under licenses that do not carry copyleft obligations. See [2] for a blog post that goes into some detail about this.
If you look at the Coretex project MAINTAINERS file [3], you will see that there are two folks listed that currently work at AWS, but no other company other than Grafana Labs today. I would love to see more diversity in maintainers for a project like this, as I think too many maintainers from any one company isn't the best for long term project sustainability.
I think if you look at the Cortex Community Meeting minutes [4], you can see that AWS folks are regularly "showing up" in healthy numbers, and working collaboratively with anyone who accepts the open invitation to participate. There have been some pretty big improvements to Coretex that have merged lately, like some of the work on parallel compaction [5, 6].
TL;DR, I think it is easy to jump to some conclusions about how things are going in a FOSS project that don't hold water if you do some cursory exploration. I think best way to know what's going on in a project is to get involved!
--
[1] the rights needed to: run the program for any purpose; to study how the program works, and modify it; to redistribute copies; to distribute copies of modified versions to others
[2] https://meshedinsights.com/2021/06/14/legally-ignoring-the-l...
[3] https://github.com/cortexproject/cortex/blob/master/MAINTAIN...
[4] https://docs.google.com/document/d/1shtXSAqp3t7fiC-9uZcKkq3m...
[5] https://aws.amazon.com/blogs/opensource/scaling-cortex-with-...
- Ask HN: How to built a HIGHLY scalable API monitoring tool?
-
The unbearable fussiness of the smart home
> [...] that feed into a prometheus -> cortex store, so I can then map them on Grafana.
I had to Google because I've never heard of any of those. Did I find the right ones?
Mine is much more primitive. My indoor temperature monitor is an ESP8266 that uploads the temperature to a simple PHP page that saves it in an sqlite DB. A cron job runs a Perl script every few minutes that extract the data for the last hour, 3 hours, 12 hours, 48 hours, and since the beginning of time and uses gnuplot to produce PNG graphs. There's a static page on my server that displays those graphs.
My outdoor temperature monitor uses a cheap AcuRite 433 MHz indoor/outdoor thermometer I bought. I have an RPi with an RTL-SDR attached spying on the communications between the AcuRite sensor outside and the AcuRite display inside using rtl_433. A script looks at the rtl_433 and finds the AcuRite sensor data and puts it in an sqlite DB. I haven't yet gotten around to making something to graph it.
The nice thing about that approach is that it was also easy to add support for other 433 MHz wireless sensors near me, such as the AcuRite fridge/freezer thermometer I have. I can also see a few assorted sensors of neighbors (temperature, humidity, soil moisture, tire pressure, wind speed, wind direction, rain, and a few other random things). If I wanted to it would be easy to add them to the DB.
When I made a wireless tipping range gauge recently. I used a 433 MHz transmitter module [1] and added a decoder [2] to rtl_433 that understands my data stream format. That gets my data into the rtl_433 output. No need to futz around with 433 MHz receiver modules which appear to be a pain in the ass [3]. An ATTiny85 counts the tips and runs the transmitter. The ATTiny85, the transmitter module, a battery holder, an RJ11 socket because the rain gauge has an RJ11 connector, a board to put those things on [4], and a small waterproof case is pretty much the complete parts list.
I think I'm going to standardize on this general approach. For things that do not have WiFi and only need to report data 433 MHz modules and custom decoders fro rtl_433 on the RPi. For things that do have WiFi, such as any future ESP projects I do, they will just use WiFi to talk to the RPi. If anything needs to get sent outside of my LAN the RPi will handle it.
The RPi is also currently controlling a space heater in my living room, getting connection data from my cable modem periodically and recording that in an sqlite DB, and serving a simple web page that lets me quickly change inputs and volume on my Denon receiver and so I'm already pretty much committed to keeping it running all the time.
[1] https://www.sparkfun.com/products/10534
[2] Decoders can be specified in a simple text file. Here's the one for my rain guage as an example:
decoder {
TimescaleDB
-
Google Cloud Spanner is now half the cost of Amazon DynamoDB
Don't forget PostgreSQL extensions. For something like a chat log, TimescaleDB (https://www.timescale.com/) can be surprisingly efficient. It will handle partitioning for you, with additional features like data reordering, compression, and retention policies.
-
How to Choose the Right MQTT Data Storage for Your Next Project
TimescaleDB{:target="_blank"}: an extension of PostgreSQL that adds time-series capabilities to the relational database model. It provides scalability and performance optimizations for handling large volumes of time-stamped data while maintaining the flexibility of a relational database.
-
Opinions and Suggestions for PostgreSQL Extension under Development
What about getting in touch with commercial organisations that have products/services based on PostgreSQL? For example Timescale, EDB, and Citus Data, or really any hosting provider that offers a managed PostgreSQL service.
-
Ask HN: It's 2023, how do you choose between MySQL and Postgres?
Friends don't let their friends choose Mysql :)
A super long time ago (decades) when I was using Oracle regularly I had to make a decision on which way to go. Although Mysql then had the mindshare I thought that Postgres was more similar to Oracle, more standards compliant, and more of a real enterprise type of DB. The rumor was also that Postgres was heavier than MySQL. Too many horror stories of lost data (MyIsam), bad transactions (MyIsam lacks transaction integrity), and the number of Mysql gotchas being a really long list influenced me.
In time I actually found out that I had underestimated one of the most important attributes of Postgres that was a huge strength over Mysql: the power of community. Because Postgres has a really superb community that can be found on Libera Chat and elsewhere, and they are very willing to help out, I think Postgres has a huge advantage over Mysql. RhodiumToad [Andrew Gierth] https://github.com/RhodiumToad & davidfetter [David Fetter] https://www.linkedin.com/in/davidfetter are incredibly helpful folks.
I don't know that Postgres' licensing made a huge difference or not but my perception is that there are a ton of 3rd party products based on Postgres but customized to specific DB needs because of the more liberalness of the PG license which is MIT/BSD derived https://www.postgresql.org/about/licence/
Some of the PG based 3rd party DBs:
Enterprise DB https://www.enterprisedb.com/ - general purpose PG with some variants
Greenplum https://greenplum.org/ - Data warehousing
Crunchydata https://www.crunchydata.com/products/hardened-postgres - high security Postgres for regulated environments
Citus https://www.citusdata.com - Distributed DB & Columnar
Timescale https://www.timescale.com/
Why Choose PG today?
If you want better ACID: Postgres
If you want more compliant SQL: Postgres
If you want more customizability to a variety of use-cases: Postgres using a variant
If you want the flexibility of using NOSQL at times: Postgres
If you want more product knowledge reusability for other backend products: Postgres
-
Help with timeseries data
TimescaleDB is Postgres with extensions to automatically partition tables for fast processing of time series data.
- Building a Cloud Database from Scratch: Why We Moved from C++ to Rust
- I would like to know your advice, I am creating an inventory control software, and I would like to use the PostgreSQL database instead of SQL Server, Could you give me your opinions of the advantages and disadvantages of using one or the other, Thank you.
-
Question: What is the Best Way to Store a ~10 Terabytes of Time Series Data?
Have you heard of timescale? https://www.timescale.com/ Seems similar to ocient but specifically for time series data.
-
Day 23: CI using timescaledb a PostgreSQL based time series database
Slowly I understood that instead of a vanilla PostgreSQL database I need to use to use Timescale which is based on PostgreSQL. I am sure others would have come to this conclusion much faster than I did.
-
Is Postgresql integration well supported in Julia?
Good question... haha I haven't really considered it. I'm no too versed in this domain and so the whole project will be a learning experience. One of the things is that it will include time-series harvest data. I was searching around for ways to implement this and found solutions like TimescaleDB and InfluxDB. Seems like also there are just some plugins that can sit on top of PostgreSQL.
What are some alternatives?
thanos - Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
mimir - Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
ClickHouse - ClickHouse® is a free analytics DBMS for big data
promscale - [DEPRECATED] Promscale is a unified metric and trace observability backend for Prometheus, Jaeger and OpenTelemetry built on PostgreSQL and TimescaleDB.
TDengine - TDengine is an open source, high-performance, cloud native time-series database optimized for Internet of Things (IoT), Connected Cars, Industrial IoT and DevOps.
loki - Like Prometheus, but for logs.
GORM - The fantastic ORM library for Golang, aims to be developer friendly
temporal_tables - Temporal Tables PostgreSQL Extension
pgbouncer - lightweight connection pooler for PostgreSQL
VictoriaMetrics - VictoriaMetrics: fast, cost-effective monitoring solution and time series database
Telegraf - The plugin-driven server agent for collecting & reporting metrics.
citus - Distributed PostgreSQL as an extension