python-mysql-replication VS Pyjion

I'm maintaining an internal change-data-capture application that uses a python library to decode mysql binlog and store the change records as json in the data lake (like Debezium). For our most busiest databases a single Cpython process couldn't process the amount of incoming changes in real time (thousands of events per second). It's not something that can be easily parallelized, as the bulk of the work is happening in the binlog decoding library (https://github.com/julien-duponchelle/python-mysql-replicati...).

So we've made it configurable to run some instances with Pypy - which was able to work through the data in realtime, i.e. without generating a lag in the data stream. The downside of using pypy was increased memory usage (4-8x) - which isn't really a problem. An actually problem that I didn't really track down was that the test suite (running pytest) was taking 2-3 times longer with Pypy than with CPython.

A few months ago I upgraded the system to run with CPython 3.11 and the performance improvements of 10-20% that come with that version now actually allowed us to drop Pypy and only run CPython. Which is more convenient and makes the deployment and configuration less complex.

doing the using Python, https://github.com/julien-duponchelle/python-mysql-replication, the recommended way of doing this

The BinLogStreamReader has several inputs that we need to retrieve. First we'll retrieve the cluster's secret with the database host/username/password and then we'll fetch the serverId we stored in S3.

From backend relational tables to data warehouses my team has mostly relied on change data capture replication. We use MySQL upstream, and historically used AWS DMS or Attunity Replicate to replicate directly to SQL server. Recently we made the switch to Snowflake, and used mostly AWS DMS to replicate CDC data to S3 (lists individual inserts, updates, deletes), and then from there use snowpipes to copy to snowflake and then a job to merge that data into the target table to get the latest state. In addition we've used this library in production https://github.com/noplay/python-mysql-replication, and still use it today for one high volume, critical data source. Generally we see data go end to end in a matter of minutes, but occasionally there are spikes in latency.

It exists, was created by microsoft employees, and is referenced in the article: https://www.trypyjion.com/

I've actually come across and started using Pyjion recently (https://github.com/tonybaloney/pyjion); how does Pypy compare, both in terms of performance and purpose? There seems to be a lot of overlap...

Python is capable of doing things at runtime that are really hard to statically compile around, such as monkeypatching methods onto existing objects. You can compile it, but it's complicated. One strategy is to use a JIT that can observe application state at runtime and then invalidate code as it becomes obsoleted by changes, but it's complicated. See pyjion for an example.

When I say "Python" I am referring to the standard CPython interpreter which most people use. But there is also PyPy, which includes a Just In Time compile that compiles selected code into machine language on the fly, as needed. pyjion is another JIT compiler that generates machine language on the fly, and you can install it with pip. Or you could work for Facebook and use Cinder. Cython, Nuitka and Pyston are other alternatives.

and if you need more speed you can try https://www.pypy.org/ or https://github.com/tonybaloney/Pyjion or https://www.pyston.org/

Finally, there is also Pyjion which based on its website is “A drop-in JIT Compiler for Python 3.10” (https://www.trypyjion.com/). We will be covering it on a separate writeup. See you next time ;-).

Though some exciting stuff happening over the next few years, Python is getting faster, has been for awhile, and stuff like Pyjion https://www.trypyjion.com/, a drop in C# powered JIT compiler is starting to approach usable. Rust and Python seem to be best buds right now, so more extension libraries in rust, a newer more approachable language than say C/C++ but with a similar speed. Sign me up!

Pyjion Easy to use, small compiler. Increase performance of our 🐌 CPython.