pushgateway VS client_python

Pushgateway

In case the Custom Metric is related to a job or a short-living script, consider using Pushgateway.

So my idea is to have a python script that buckets data and then publishes it again using pushtogateway: https://github.com/prometheus/pushgateway

Is the cron job with a script for getting metric and pushing via https://github.com/prometheus/pushgateway the right way for this task?

The push gateway is a relatively simple service. There’s one main entry point file (main.go) and an api file (api.go) both at 279 lines log. There’s probably a dozen or so api endpoints, but the key one is the one where you post metrics, defined here

We are currently evaluating timeseries databases to build an internal tool for DORA metrics and just looking at prometheus. As tracking releases is more of a push based use case, I looked at pushgateway and the docs stated the following "The Pushgateway is not an event store. [...] tracking something like release events has to happen with some event-logging framework." here

Checked this out? https://github.com/prometheus/pushgateway

While I somehow understand Prometheus idea that pull is easier to scale than push I've had a bad luck with it.

First of all Prometheus doesn't even consider monitoring of long-running jobs other than pull way (which didn't make sense for me). There is push gateway [0] but clients libraries seem to consider it only for short-lived jobs where you can send the metrics at the end [1]. It seems I couldn't "push" from long living jobs trivially

Second when using it for example with django you have to be careful with how you handle multiprocessing that UWSGI/gunicorn does, see [2] - it has bitten me at leas once.

Comparing to push model where I can just push metrics to [3] statsd_exporter directly and be done with it, but support for statsd is lacking both in terms of frameworks (everyone seems to be migrating to native clients...) and functionality (you've to do labeling basically manually [4])

To sum up: Prometheus is really great when it works, until you try to go off-track (intentionally or not, see django [2]) then you see its all undiscovered and immature landscape

[0] https://github.com/prometheus/pushgateway

Here you go: https://stackoverflow.com/questions/75652326/celery-spawn-si...

Plus some adjacent discussion on GitHub: https://github.com/prometheus/client_python/issues/902

Hope that helps!

Prometheus, which is also a CNCF open source project, collects metrics data by scraping HTTP endpoints and then stores that data in a time series database that uses a multidimensional model. It’s a powerful tool for gathering metrics about your application and it also includes alerting functionality that you can use to notify your teams when issues come up. Prometheus includes a client library for Python.

We have a java one but the principle is the same. Install the Prometheus client ( https://github.com/prometheus/client_python) ,create the metrics you want, then push jmx settings to Prometheus.

The code above is copied from the official documentation of prometheus_client which simply creates a new metric named request_processing_seconds that measures the time spent on that particular request. We'll cover other types of metrics later in this post.

Just use the client? https://github.com/prometheus/client_python

I've experimented with the official Prometheus python client, i really really like the way they use decorators to instrument. I've tried to measure latency with multiple types of metrics (histogram, & summary), i see the value in both of them, but the one that between fits my objective is the histogram metric type. Great!

Now is a good time to expand your event loop by adding metrics collection of the event handler functions and also use that endpoint as a liveness probe. E.g. https://github.com/prometheus/client_python just add the event handled, success/error and the duration as a histogram (look for examples of tracking http requests served)

I've spent along time trying to figure out a bug that I'm facing while using Prometheus from its python client.

The official Prometheus Python client library makes this easy, no need to worry about the export file format.