vector
tracing
Our great sponsors
vector | tracing | |
---|---|---|
95 | 52 | |
16,366 | 4,919 | |
4.8% | 2.9% | |
9.9 | 8.1 | |
6 days ago | 2 days ago | |
Rust | Rust | |
Mozilla Public License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
vector
- FLaNK AI Weekly 18 March 2024
- Vector: A high-performance observability data pipeline
-
Hacks to reduce cloud spend
we are doing something similar with OTEL but we are looking at using https://vector.dev/
-
About reading logs
We don't pull logs, we forward logs to a centralized logging service.
-
Self hosted log paraer
opensearch - amazon fork of Elasticsearch https://opensearch.org/docs/latestif you do this an have distributed log sources you'd use logstash for, bin off logstash and use vector (https://vector.dev/) its better out of the box for SaaS stuff.
-
creating a centralize syslog server with elastic search
I have done something similar in the past: you can send the logs through a centralized syslog servers (I suggest syslog-ng) and from there ingest into ELK. For parsing I am advice to use something like Vector, is a lot more faster than logstash. When you have your logs ingested correctly, you can create your own dashboard in Kibana. If this fit your requirements, no need to install nginx (unless you want to use as reverse proxy for Kibana), php and mysql.
-
Show HN: Homelab Monitoring Setup with Grafana
I think there's nothing currently that combines both logging and metrics into one easy package and visualizes it, but it's also something I would love to have.
Vector[1] would work as the agent, being able to collect both logs and metrics. But the issue would then be storing it. I'm assuming the Elastic Stack might now be able to do both, but it's just to heavy to deal with in a small setup.
A couple of months ago I took a brief look at that when setting up logging for my own homelab (https://pv.wtf/posts/logging-and-the-homelab). Mostly looking at the memory usage to fit it on my synology. Quickwit[2] and Log-Store[3] both come with built in web interfaces that reduce the need for grafana, but neither of them do metrics.
- [1] https://vector.dev
-
Retaining Logs generated by service running in pod.
Log to stdout/stderr and collect your logs with a tool like vector (vector.dev) and send it to something like Grafana Loki.
-
Lightweight logging on RPi?
I would recommend that you run vector as a systems service so you don't have to worry about managing it. Here is a basic config to do that - https://github.com/vectordotdev/vector/blob/master/distribution/systemd/vector.service .
-
Monitoring traefik access logs easily
You could have a look at Grafana Loki, it's easy to run (single binary for a small setup). Shipping your logs can be done by Promtail or something like Vector. They're both lightweight log shippers with support for Loki.
tracing
-
Decrusting the tracing crate [video] by Jon Gjengset
The video description is as follows:
In this stream, we peel back the crust on the tracing crate — https://github.com/tokio-rs/tracing/ — and explore its interface, structure, and mechanisms. We talk about spans, events, their attributes and fields, and how to think about them in async code. We also dig into what subscribers are, how they pick up events, and how you can construct your own subscribers through the layer abstraction. For more details about tracing, see https://docs.rs/tracing/latest/tracing/.
-
Vendor lock-in is in the small details
> What's been your biggest issues around ergonomics/amenities for OpenTelemetry?
I can't speak generally, but in the Rust ecosystem the various crates don't play well together. Here's one example: <https://github.com/tokio-rs/tracing/issues/2648> There are four crates involved (tracing-attributes, tracing-opentelemetry, opentelemetry, and opentelemetry-datadog) and none of them fit properly into any of the others.
-
Grimoire - A recipe management application.
The tracing (logging) mechanism in an asynchronous codebase (tracing).
-
How easy is it to swap out your async runtime?
Tracing is Tokio's alternative for async code.
-
Hey Rustaceans! Got a question? Ask here (27/2023)!
At a technical level, in Rust, both [tracing]https://crates.io/crates/tracing) and log are entire ecosystems (though for the latter at least there's also third party logging frameworks), and there's at least a bridge from log to tracing.
-
How can I write a tracing subscriber that saves to a database?
I am using https://github.com/tokio-rs/tracing for logging purposes in my application. I would like to develop a feature wherein logs should be saved to a database table (via sea-orm). Something similar is this, but it does not solve my needs fully.
-
A locking war story
I've used the tracing infrastructure with tracing_flame to profile some hot paths in async code: https://github.com/tokio-rs/tracing/tree/master/tracing-flame
-
I was wrong about rust
Oh nice! IIRC when I checked, it was the Unicode tables that smashed the code size. I recently hit the same issue with the tracing crate, where a crate feature (for env var filtering) pulled in regex and my binary was suddenly 1MB bigger.
-
Debugging and profiling embedded applications.
I know about tools such as tracing, jaeger or tracy. While having a complete tracing could be a potential solution, these tools don't work with no_std.
-
Custom Axum Logging for Routes?
tracing by itself only outputs log data, you need to consume them in a subscriber, the tracing-subscriber crate exists for this. (example)
What are some alternatives?
graylog - Free and open log management
log4rs - A highly configurable logging framework for Rust
Fluentd - Fluentd: Unified Logging Layer (project under CNCF)
slog - Structured, contextual, extensible, composable logging for Rust
agent - Vendor-neutral programmable observability pipelines.
env_logger - A logging implementation for `log` which is configured via an environment variable.
syslog-ng - syslog-ng is an enhanced log daemon, supporting a wide range of input and output methods: syslog, unstructured text, queueing, SQL & NoSQL.
log - Logging implementation for Rust
OpenSearch - 🔎 Open source distributed and RESTful search engine.
opentelemetry-rust - The Rust OpenTelemetry implementation
qryn - qryn is a polyglot, high-performance observability framework for ClickHouse. Ingest, store and analyze logs, metrics and telemetry traces from any agent supporting Loki, Prometheus, OTLP, Tempo, Elastic, InfluxDB and many more formats and query transparently using Grafana or any other compatible client.
flamegraph - Easy flamegraphs for Rust projects and everything else, without Perl or pipes <3