python-mysql-replication
direnv
python-mysql-replication | direnv | |
---|---|---|
5 | 160 | |
2,268 | 11,776 | |
- | 1.6% | |
9.1 | 8.7 | |
3 days ago | 15 days ago | |
Python | Go | |
- | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
python-mysql-replication
-
Is anyone using PyPy for real work?
I'm maintaining an internal change-data-capture application that uses a python library to decode mysql binlog and store the change records as json in the data lake (like Debezium). For our most busiest databases a single Cpython process couldn't process the amount of incoming changes in real time (thousands of events per second). It's not something that can be easily parallelized, as the bulk of the work is happening in the binlog decoding library (https://github.com/julien-duponchelle/python-mysql-replicati...).
So we've made it configurable to run some instances with Pypy - which was able to work through the data in realtime, i.e. without generating a lag in the data stream. The downside of using pypy was increased memory usage (4-8x) - which isn't really a problem. An actually problem that I didn't really track down was that the test suite (running pytest) was taking 2-3 times longer with Pypy than with CPython.
A few months ago I upgraded the system to run with CPython 3.11 and the performance improvements of 10-20% that come with that version now actually allowed us to drop Pypy and only run CPython. Which is more convenient and makes the deployment and configuration less complex.
-
Why Binlog size grows drastically when isolation level set to "Repeatable Read" & When isolation level set to "Read Committed" the size of Binlog file reduces ?
doing the using Python, https://github.com/julien-duponchelle/python-mysql-replication, the recommended way of doing this
-
How to Use BinLogs to Make an Aurora MySQL Event Stream
The BinLogStreamReader has several inputs that we need to retrieve. First we'll retrieve the cluster's secret with the database host/username/password and then we'll fetch the serverId we stored in S3.
-
How is everyone ingesting backend relational data?
From backend relational tables to data warehouses my team has mostly relied on change data capture replication. We use MySQL upstream, and historically used AWS DMS or Attunity Replicate to replicate directly to SQL server. Recently we made the switch to Snowflake, and used mostly AWS DMS to replicate CDC data to S3 (lists individual inserts, updates, deletes), and then from there use snowpipes to copy to snowflake and then a job to merge that data into the target table to get the latest state. In addition we've used this library in production https://github.com/noplay/python-mysql-replication, and still use it today for one high volume, critical data source. Generally we see data go end to end in a matter of minutes, but occasionally there are spikes in latency.
- Robust data transfer mechanism?
direnv
-
Show HN: Dotenv, if it is a Unix utility
I think direnv already does a good job in this space, and it's already available in your package manager.
https://direnv.net/
-
Mise is a polyglot tool version manager
I switched from asdf to mise after a comment on lobste.rs[1] suggested I do so a few months ago, and I have been very happy with it.
It sands off some of asdf's sharp UI edges and provides a somewhat larger but still reasonable feature set; I've also replaced most of my direnv[2] usage with it.
The mise -> asdf comparison page is useful[3]
1: https://lobste.rs/s/66uxbj/how_love_homebrew#c_mvmsjp
2: https://direnv.net/
3: https://mise.jdx.dev/dev-tools/comparison-to-asdf.html
-
Nix-direnv is a quality of life improvement
I also made the export diff configurable, motivated by this post: https://github.com/direnv/direnv/pull/1233
- Direnv – Unclutter Your .profile
-
Conditional Git Configuration
Nice.
For years I've been using [direnv](https://direnv.net/) for this, setting environment variables which git picks up. This looks like a more feature complete equivalent, although to be honest I only really need switching of committer email and the SSH key used.
- FLaNK 25 December 2023
-
Development Environments with Guix, similar to devenv.sh
Direnv, for the uninitiated, loads and unloads environment variables when directories are entered and exited. Under every project folder there is a `$PROJ_DIR/.envrc` which contains:
-
Emacs Advent Calendar 9: devdocs, code-cells, dREPL, etc.
buffer-env: A pure-Elisp version of the direnv utility. Useful to make Emacs aware of Python virtualenvs (which, judging by the questions posted here, is unfortunately still a complication for a lot of people). Similar to (and inspired by) envrc, but doesn't require the direnv program.
-
golang cli vs env var in windows?
You can look at direnv to see this in action as they wrote shell hooks that get loaded into the shell profile and are executed on every prompt. https://direnv.net/
-
Nix Survival Mode: macOS upgrades won't break Nix anymore
Yes, most Nix users employ https://direnv.net or the equivalent for your IDE of choice. Emacs for instance has https://github.com/purcell/envrc which set per-buffer variables.
What are some alternatives?
AWS Data Wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
spaceship-prompt - :rocket::star: Minimalistic, powerful and extremely customizable Zsh prompt
PyMySQL - MySQL client library for Python
Pipenv - Python Development Workflow for Humans.
PonyORM - Pony Object Relational Mapper
lorri - Your project's nix-env
sparc-curation - code and files for SPARC curation workflows
Home Manager using Nix - Manage a user environment using Nix [maintainer=@rycee]
preshed - 💥 Cython hash tables that assume keys are pre-hashed
Neovim-from-scratch - 📚 A Neovim config designed from scratch to be understandable
mycli - A Terminal Client for MySQL with AutoCompletion and Syntax Highlighting.
Poetry - Python packaging and dependency management made easy