pprof VS opentracing-javascript

The pprof format is not tied to Go. From my understanding, it's used within Google across multiple languages. The format is defined in the pprof repository[0], and the visualization tool is source-language agnostic. I've seen libraries in numerous languages (e.g. Python, Java) to publish profiles in pprof format. This is an indicator the pprof format has become de-facto. Grafana Pyroscope[1] is a tool that's capable of parsing the pprof format, agnostic to the source programming language, and has instructions for Go, Java, Python, Ruby, node.js, Rust, and .NET.

My understanding is that you're searching for a combination of the profiles, metrics, and tracing. Caddy supports all 3.

[0] https://github.com/google/pprof/blob/main/doc/README.md

[1] https://grafana.com/docs/pyroscope/latest/

metrics and tracing need to be manually enabled (for now, perhaps)

Because we couldn't identify the issue using the results we got from Callgrind, we reached for another profiler, gperftools. It's a sampling profiler and therefor it has a smaller impact on the application's performance in exchange for less accurate call statistics. After filtering out the unimportant parts and visualizing the rest with pprof, it was evident that something strange was happening with the send function. It took only 71 milliseconds with the previous implementation and more than 900 milliseconds with the new implementation of our Bolt server. It was very suspicious, but based on Callgrind, its cost was almost the same as before. We were confused as the two results seemed to conflict with each other.

github.com - google/pprof

The application performances in a specific code path (e.g. gdb, pprof, …).

pprof is https://github.com/google/pprof, it's a very useful tool in golang , and really really really convenient

Go also has pretty good out of the box profiling (pprof[0]) and third-party runtime debugging (delv[1]) that can be used both remotely and local.

These tools also have decent editor integration and can be use hand in hand:

https://blog.jetbrains.com/go/2019/04/03/profiling-go-applic...

https://blog.jetbrains.com/go/2020/03/03/how-to-find-gorouti...

[0] https://github.com/google/pprof

[1] https://github.com/go-delve/delve

And what we do in Google Cloud is that we still use the pprof. But it's a kind of forked version of the pprof because the visualization part is totally different. So we give that tool as the Cloud Profiler. So that is the product name. And then, the difference between the pprof and a Cloud Profiler is that Cloud Profiler provides the agent library for each famous programming language such as Java, Python, Node.js, and Go. And then what you need to do is to just write 5 to 10 lines of code in a new application. That launches the profile agent in your application as a subsidiary thread of the main thread. And then, that thread periodically collects the profile data of the application and then sends that data back to Google Cloud and the Cloud Profiler.

gprftools (https://github.com/gperftools/gperftools) can be easily plugged in using LD_PRELOAD and signal, and has nice go implemented visualization tool https://github.com/google/pprof.

Spring Spring web framework Spring WebFlux Reactive REST Services Spring Data R2DBC a specification to integrate SQL databases using reactive drivers Redisson Redis Java Client Zipkin open source, end-to-end distributed tracing Spring Cloud Sleuth auto-configuration for distributed tracing Prometheus monitoring and alerting Grafana for to compose observability dashboards with everything from Prometheus Kubernetes automating deployment, scaling, and management of containerized applications Docker and docker-compose Helm The package manager for Kubernetes Flywaydb for migrations

Elasticsearch client for Go RabbitMQ Go RabbitMQ Client Library Jaeger open source, end-to-end distributed tracing Prometheus monitoring and alerting Grafana for to compose observability dashboards with everything from Prometheus Echo web framework Kibana is user interface that lets you visualize your Elasticsearch Docker and docker-compose Kubernetes K8s Helm The package manager for Kubernetes

PostgeSQL as event store database Kafka as messages broker gRPC Go implementation of gRPC Jaeger open source, end-to-end distributed tracing Prometheus monitoring and alerting Grafana for to compose observability dashboards with everything from Prometheus MongoDB MongoDB database Elasticsearch Elasticsearch client for Go. Echo web framework Kibana Kibana is data visualization dashboard software for Elasticsearch Migrate for migrations

OpenTracing was an open-source project aimed at providing vendor-neutral APIs and instrumentation for distributed tracing. In distributed cloud-native applications, it is difficult for engineering teams to see how requests are performing across services. And that’s where distributed tracing comes into the picture.

OpenTracing is now archived, and it is suggested to migrate to OpenTelemetry.

In this article let's try to create closer to real world Event Sourcing CQRS microservice using: 🚀👨‍💻🙌 EventStoreDB The database built for Event Sourcing gRPC Go implementation of gRPC MongoDB Web and API based SMTP testing Elasticsearch Elasticsearch client for Go. Jaeger open source, end-to-end distributed tracing Prometheus monitoring and alerting Grafana for to compose observability dashboards with everything from Prometheus swag Swagger for Go Echo web framework Kibana Kibana is user interface that lets you visualize your Elasticsearch

I agree; but I think it'll take years, if ever, to change this culture.

Logging is a byproduct of a past time; everything is a file, stdout is a file, lets persist that file, now we have multiple replicas, lets collect the file into a multi-terabyte searchable database.

The biggest downside: Its WILDLY expensive. Large orgs often have an entire team dedicated to maintaining logging (ELK) infrastructure. This price-tag inevitably leads to bikeshedding on backend teams about how to "reduce the amount we're logging" or "cleaning up the logs" or "structuring them to be more useful".

Outside of development, they are so rarely useful. Yet inevitably someone will say: "you're just not structuring your logs correctly." Maybe that's true; similarly, I don't find vim to be a highly productive editing experience. Maybe I just don't have the thousands of extensions it would take to make it so. Or maybe You're stuck in the past and ignoring two decades of tooling improvement. Both can be true.

I'm phrasing this as a false dichotomy, because in many teams: it is. Logging is easy; its built-in to most languages; so devs log. The information we need is in the system; its a needle in a haystack, but the needle is there. We log when a request comes in, when it hits pertinent functions, when its finished, how its finished, the manager says: "just look at the logs". Instead of "what better tooling can we make an investment in so future investigations of this nature don't take a full day."

For starters: Tracing. Tracing systems should be built-in to EVERY LANGUAGE, just like console.log. We have a standard [1] sponsored by the Linux Foundation and supported by every major trace ingestion system. This is not a problem of camps and proprietary systems; its a problem of culture. I should be able to call a nodejs stdlib function at startup, specify where I want traces to go, sampling rate, etc, and immediately get every single function call instrumented. Its literally just highly-structured-by-default logging! Dump the spans to stdout by default! Our log ingestion systems can read each line, determine if its a span, if so route to trace ingestion, otherwise route to log ingestion.

This is a critical step because it asserts that tracing is actually a very powerful tool that everyone needs to learn, like logging. Everyone knows about logging. Why? console.log. Its there, it gets used. Tracing right now is relegated to a subculture of "advanced diagnostics"; you gotta adopt a tracing provider, bring in dependencies, learn each implementation of OpenTracing, authenticate to send traces over HTTP... as a community, we should normalize just "dump traces like you dump logs, to stdout", have a formatter to make them nice to use in development, and now all that instrumentation work that any dev is capable of utilizing (just like console.log) "just works" in production.

[1] https://opentracing.io/

Service mesh - ssup2ket services run on service mesh for detailed traffic control and easy monitoring. Service mesh is applied through Istio. Istio uses OpenTracing for easy request tracing between multiple services.