oteps
veneur
Our great sponsors
| oteps | veneur | |
|---|---|---|
| 4 | 2 | |
| 316 | 1,714 | |
| 1.9% | 0.5% | |
| 5.3 | 3.5 | |
| 9 days ago | about 1 month ago | |
| Makefile | Go | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
oteps
-
OpenTelemetry in 2023
Oh nice, thank you (and also solumos) for the links! It looks like oteps/pull/171 (merged June 2023) expanded and superseded the opentelemetry-proto/pull/346 PR (closed Jul 2022) [0]. The former resulted in merging OpenTelemetry Enhancement Proposal 156 [1], with some interesting results especially for 'Phase 2' where they implemented columnar storage end-to-end (see the Validation section [2]):
* For univariate time series, OTel Arrow is 2 to 2.5 better in terms of bandwidth reduction ... and the end-to-end speed is 3.1 to 11.2 times faster
* For multivariate time series, OTel Arrow is 3 to 7 times better in terms of bandwidth reduction ... Phase 2 has [not yet] been .. estimated but similar results are expected.
* For logs, OTel Arrow is 1.6 to 2 times better in terms of bandwidth reduction ... and the end-to-end speed is 2.3 to 4.86 times faster
* For traces, OTel Arrow is 1.7 to 2.8 times better in terms of bandwidth reduction ... and the end-to-end speed is 3.37 to 6.16 times faster
[0]: https://github.com/open-telemetry/opentelemetry-proto/pull/3...
[1]: https://github.com/open-telemetry/oteps/blob/main/text/0156-...
[2]: https://github.com/open-telemetry/oteps/blob/main/text/0156-...
-
Grafana Phlare, open source database for continuous profiling at scale
https://github.com/open-telemetry/oteps/issues/139
It takes a lot of time and effort to bake a cross-vendor cross-language standard.
-
Faster Protocol Buffers
This. The statelessness of the OTLP is by design. I did consider stateful designs with e.g. shared state dictionary compression but eventually chose not to, so that the intermediaries can remain stateless.
An extension to OTLP that uses shared state (and columnar encoding) to achieve more compact representation and is suitable for the last network leg in the data delivery path has been proposed and may become a reality in the future: https://github.com/open-telemetry/oteps/pull/171
veneur
-
OpenTelemetry in 2023
This was the idea behind Stripe's Veneur project - spans, logs, and metrics all in the same format, "automatically" rolling up cardinality as needed - which I thought was cool but also that it would be very hard to get non-SRE developers on board with when I saw a talk about it a few years ago.
https://github.com/stripe/veneur
-
Launch HN: Opstrace (YC S19) – open-source Datadog
One pain point with Prometheus is that is has relatively weak support for quantiles, histograms, and sets[1]:
- Histograms require manually specifying the distribution of your data, which is time-consuming, lossy, and can introduce significant error bands around your quantile estimates.
- Quantiles calculated via the Prometheus "summary" feature are specific to a given host, and not aggregatable, which is almost never what you want (you normally want to see e.g. the 95th percentile value of request latency for all servers of a given type, or all servers within a region). Quantiles can be calculated from histograms instead, but that requires a well-specified histogram and can be expensive at query time.
- As far as I know, Prometheus doesn't have any explicit support for unique sets. You can compute this at query time, but persisting and then querying high-cardinality data in this way is expensive.
Understanding the distribution of your data (rather than just averages) is arguably the most important feature you want from a monitoring dashboard, so the weak support for quantiles is very limiting.
Veneur[2] addresses these use-cases for applications that use DogStatsD[3] by using clever data structures for approximate histograms[4] and approximate sets[5], but I believe its integration with Prometheus is limited and currently only one-way - there is a CLI app to poll Prometheus metrics and push them into Veneur, but there's no output sink for Veneur to write to Prometheus (or expose metrics for a Prometheus instance to poll).
It would be extremely useful to have something similar for Prometheus, either by integrating with Veneur or implementing those data structures as an extension to Prometheus.
[1] https://prometheus.io/docs/practices/histograms/
[2] https://github.com/stripe/veneur
[3] https://docs.datadoghq.com/developers/dogstatsd/
[4] https://github.com/stripe/veneur#approximate-histograms
[5] https://github.com/stripe/veneur#approximate-sets
What are some alternatives?
zipkin-api - Zipkin's language independent model and HTTP Api Definitions
opstrace - The Open Source Observability Distribution
b3-propagation - Repository that describes and sometimes implements B3 propagation
cortex - A horizontally scalable, highly available, multi-tenant, long term Prometheus.
odigos - Distributed tracing without code changes. 🚀 Instantly monitor any application using OpenTelemetry and eBPF
Cortex - Cortex: a Powerful Observable Analysis and Active Response Engine
exp-lazyproto - Experimental fast implementation of Protobufs in Go
loki - Like Prometheus, but for logs.
community - OpenTelemetry community content
influxdb-apply - Define InfluxDB users and databases with a yaml file.
terraform-aws-jaeger - Terraform module for Jeager