Our great sponsors
-
cryptography
cryptography is a package designed to expose cryptographic primitives and recipes to Python developers.
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
first, I did see a correlation between an endpoint being heavily hit in a given time window, and an increase of memory usage that didn't went down afterwards. The endpoint didn't do much so I went through every instruction - is a global variable appended indefinitely ? Is a cache decorator growing without a limit set ? Do I use a 3rd party that has a known issue ? Turns out, it was using cryptography, so I looked up known issues. Saw an issue about a leak when using load_pem_x509_certificate https://github.com/pyca/cryptography/issues/4833 - which I used ! I could fortunately just upgrade the library
second time was nastier. I used https://github.com/bloomberg/memray to try to spot it - that's the tool you should try out. You load your service through memray, and it will get you some stats that you can export as a flamegraph. I can't really afford to make it run on production so I ran it in a docker image and repeatedly ran the scenario I thought was responsible. Didn't find anything. I know what I did wrong: I assumed one particular codepath was the problem. If would have find the issue if I had a really complete scenario that covers broadly every possible endpoint and condition. Can't blame memray, that tool is really promising.