Our great sponsors
-
-
self-hosted
Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
blacklite
"Fast as internal ring buffer" Logback/Log4J2 appender using SQLite with zstandard dictionary compression and rollover.
-
Technically, you can also use Skywalking for log aggregation, but personally the setup isn't as great and their log view UI is a bit awkward (e.g. it's not easy to preview all logs for a particular service/instance in a file-like view), see the demo: https://skywalking.apache.org/
For logs in particular, Graylog feels reasonably sane, since it has a similarly "manageable" amount of components, for a configuration example see: https://docs.graylog.org/docs/docker#settings
Contrast that to some of the more popular solutions out there, like Sentry, which gets way more complicated really quickly: https://github.com/getsentry/self-hosted/blob/master/docker-...
For most of the people who have to deal with self-hosted setups where you might benefit from something like tracing or log shipping, actually getting the platform up and running will be an uphill battle, especially if not everyone sees the value in setting something like this up, or setting aside enough resources for it. Sometimes people will be more okay with having no idea why a system goes down randomly, rather than administering something like this constantly and learning new approaches, instead of just rotating a bunch of files.
For others, there are no such worries, because they can open their wallets (without worrying about certain regulations and where their data can be stored, hopefully) and have some cloud provider give them a workable solution, so they just need to integrate their apps with some agent for shipping the information.
For others yet, throwing the requirement over to some other team who's supposed to provide such platform components for them is also a possibility.
Technically, you can also use Skywalking for log aggregation, but personally the setup isn't as great and their log view UI is a bit awkward (e.g. it's not easy to preview all logs for a particular service/instance in a file-like view), see the demo: https://skywalking.apache.org/
For logs in particular, Graylog feels reasonably sane, since it has a similarly "manageable" amount of components, for a configuration example see: https://docs.graylog.org/docs/docker#settings
Contrast that to some of the more popular solutions out there, like Sentry, which gets way more complicated really quickly: https://github.com/getsentry/self-hosted/blob/master/docker-...
For most of the people who have to deal with self-hosted setups where you might benefit from something like tracing or log shipping, actually getting the platform up and running will be an uphill battle, especially if not everyone sees the value in setting something like this up, or setting aside enough resources for it. Sometimes people will be more okay with having no idea why a system goes down randomly, rather than administering something like this constantly and learning new approaches, instead of just rotating a bunch of files.
For others, there are no such worries, because they can open their wallets (without worrying about certain regulations and where their data can be stored, hopefully) and have some cloud provider give them a workable solution, so they just need to integrate their apps with some agent for shipping the information.
For others yet, throwing the requirement over to some other team who's supposed to provide such platform components for them is also a possibility.
I've played with trying to create an idiomatic wrapper for Go precisely because of this reason, but then didn't find time to continue with this.
https://github.com/henvic/tel
I see! Yeah, this is one where where otel-go is a lot harder to use, but it's something the SIG is looking at. A coworker of mine is helping drive a design that's sort of an "easy button" to configure all the things with the least-surprising defaults[0] and we're seeing how people like it in our SDK distribution that uses it[1]. I hope that sometime soon we'll have the design polished-up enough to get merged in. Like most OSS projects, it'll take some time but I'm confident we can get it done.
The main challenge is that there's a large variety of use cases to fulfill (e.g., someone wants custom context propagation, a custom span processor, and export over HTTP+json but not HTTP+protobuf) and today the answer to that is that you have to pull in all the libraries for all the things you need. It's a lot more energy you need to expend to get started with all of this than it needs to be.
As for logging support in the Go SDK, it's frozen mostly just due to lack of bandwidth and a need to finish what's already been started. Metrics have proven to be much more difficult and time-consuming to implement correctly across all languages, with Go being impacted harder than other languages (e.g., Python and .NET). I think you can expect logging integrations in the near-ish future though.
This is great feedback. I'll pass it on folks who haven't seen it. Thank you! And please feel free to file issues about all the things that rub you the wrong way
[0]: https://github.com/open-telemetry/opentelemetry-go-contrib/p...
[1]: https://github.com/honeycombio/honeycomb-opentelemetry-go
I see! Yeah, this is one where where otel-go is a lot harder to use, but it's something the SIG is looking at. A coworker of mine is helping drive a design that's sort of an "easy button" to configure all the things with the least-surprising defaults[0] and we're seeing how people like it in our SDK distribution that uses it[1]. I hope that sometime soon we'll have the design polished-up enough to get merged in. Like most OSS projects, it'll take some time but I'm confident we can get it done.
The main challenge is that there's a large variety of use cases to fulfill (e.g., someone wants custom context propagation, a custom span processor, and export over HTTP+json but not HTTP+protobuf) and today the answer to that is that you have to pull in all the libraries for all the things you need. It's a lot more energy you need to expend to get started with all of this than it needs to be.
As for logging support in the Go SDK, it's frozen mostly just due to lack of bandwidth and a need to finish what's already been started. Metrics have proven to be much more difficult and time-consuming to implement correctly across all languages, with Go being impacted harder than other languages (e.g., Python and .NET). I think you can expect logging integrations in the near-ish future though.
This is great feedback. I'll pass it on folks who haven't seen it. Thank you! And please feel free to file issues about all the things that rub you the wrong way
[0]: https://github.com/open-telemetry/opentelemetry-go-contrib/p...
[1]: https://github.com/honeycombio/honeycomb-opentelemetry-go
> My God. Logging in protobuf?
Yes, or any other data format and/or transport protocol.
I'm surprised this is up for debate.
> Logging is the lowest of all debugging utilities - its the first thing you ever do writing software - “hello world”. And, while I admire structural logging, the truth is printing strings remains (truly) the lowest common denominator across software developers.
This sort of comment is terribly miopic. You can have a logging API, and then configure your logging to transport the events anywhere, any way. This is a terribly basic feature and requirement, and one that comes out of the box with some systems. Check how SLF4J[1] is pervasive in Java, and how any SLF4J implementation offers logging to stdout or a local file as a very specific and basic usecase.
It turns out that nowadays most developers write software that runs on many computers that aren't stashed over or under their desks, and thus they need efficient and convenient ways to check what's happening either in a node or in all deployments.
[1] https://www.slf4j.org/
> I'm surprised this is up for debate.
I looked into logging in protobuf when I was seeing if there was a better binary encoding for ring-buffer logging, along the same lines as nanolog:
https://tersesystems.com/blog/2020/11/26/queryable-logging-w...
What I found was that it's typically not the binary encoding vs string encoding that makes a difference. The biggest factors are "is there a predefined schema", "is there a precompiler that will generate code for this schema", and "what is the complexity of the output format". With that in mind, if you are dealing with chaotic semi-structured data, JSON is pretty good, and actually faster than some binary encodings:
https://github.com/eishay/jvm-serializers/wiki/Newer-Results...
That's why you have otel logging: https://github.com/open-telemetry/opentelemetry-specificatio...
That's why you have otel logging: https://github.com/open-telemetry/opentelemetry-specificatio...
Related posts
- Fluent Bit with ECS: Configuration Tips and Tricks
- dash0hq/otelbin: Web-based tool to facilitate OpenTelemetry collector configuration editing and verification
- Slog: Zero-dependency structured logging in Go
- How to integrate sentry to Nextjs application
- Methods and processes for reduce bugs in production