Tracing and visualizing the Python GIL with perf and VizTracer

• per4m

  4 47 0.0 Python

  Profiling and tracing information for Python using viztracer and perf, the GIL exposed.

VizTracer is a Python tracer that can visualize what your program does in the browser. Let us run it on a slightly more advanced example to see what it looks like.

• viztracer

  5 4,325 7.7 Python

  VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

Of course, you need to build some tooling around this, to really see what is happening. But one can imagine this output can be easily parsed and this is what per4m does. However, before we go there, I'd first like to visualize the flow of a slightly more advanced program using VizTracer.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Apache Arrow

  75 13,480 10.0 C++

  Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

  

Automatically detect which C function does not release the GIL (basically autodetect https://github.com/vaexio/vaex/pull/1114 https://github.com/apache/arrow/pull/7756 )

• gil_load

  1 109 0.0 Cython

  Utility for measuring the fraction of time the CPython GIL is held

Note that the per4m perf2trace gil subcommand also gives a gil_load like output. From this output, we see that both threads are waiting for the GIL approximately 50% of the time, as expected.

• h5py

  5 1,996 8.6 Python

  HDF5 for Python -- The h5py package is a Pythonic interface to the HDF5 binary data format.

  

Apply these to more issues, like in https://github.com/h5py/h5py/issues/1516

• CPython

  1,313 59,531 10.0 Python

  The Python programming language

  

Looking at the output of perf script (where we add --no-inline for performance reasons), we get a load of information. Looking at a state change event, we can now see that take_gil was called!

• gilstats.py

  1 10 0.0 Python

  A utility for dumping per-thread statistics for CPython GIL using eBPF

Note: We also see that pthread_cond_timedwait is called, this is what https://github.com/sumerc/gilstats.py uses for a eBPF tool, in case you are interested in other mutexes.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project