asyncpg
pgbouncer
Our great sponsors
asyncpg | pgbouncer | |
---|---|---|
15 | 34 | |
6,596 | 2,627 | |
1.0% | 3.0% | |
7.2 | 8.8 | |
5 days ago | 5 days ago | |
Python | C | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
asyncpg
-
Ask HN: Is Python async/await some kind of joke?
- SqlAlchemy/asyncpg => you can’t use it if you’re using PgBouncer (necessary most of the time with Postgres) in transaction mode? What?? https://github.com/MagicStack/asyncpg/issues/1058
-
Ruby Outperforms C: Breaking the Catch-22
This pure Python library claims quite fabulous performance: https://github.com/MagicStack/asyncpg
I believe it because that team have done lots of great stuff but I haven't used it, I just remembered thinking it was interesting the performance was so good. Not sure how related it is to running on the asyncio loop (or which loop they used for benchmarks).
-
PgBouncer is useful, important, and fraught with peril
what a great post, we have had a ton of issues with users using pgbouncer and it's not because things are "broken" per se, it's just the situation is very complicated, and pgbouncer's docs are also IMO in need of updating to be more detailed and in a few critical cases less misleading, specifically the prepared statements docs.
This blog post refers to this misleading nature at https://jpcamara.com/2023/04/12/pgbouncer-is-useful.html#pre... .
> PgBouncer says it doesn’t support prepared statements in either PREPARE or protocol-level format. What it actually doesn’t support are named prepared statements in any form.
That's also not really accurate. You can use a named prepared statement just fine in transaction mode. start a transaction (so you aren't in autocommit), use a named statement, works fine. you just can't use it again in another transaction, because it will be "gone" (more accurately, "unmoored" - might be in your session, might be in someone else's session). Making things worse, when the prepared statement is "unmoored", its name can then conflict with another client attempting to use the same name.
so to use named prepared statements, you can less ideally name them with random strings to avoid conflicts, or you can DEALLOCATE the prepared statement(s) you used at the end of your transaction. for our users that use asyncpg, we have them use a uuid for prepared statements to avoid these name conflicts (asyncpg added this feature for us here: https://github.com/MagicStack/asyncpg/issues/837). however, they can just as well use DEALLOCATE ALL, set this as their `server_reset_query`, and then so that happens in transaction mode, also set `server_reset_query_always`, so that it's called at the end of transactions. Where pgbouncer here IMO entirely misleadingly documents this as "This setting is for working around broken setups that run applications that use session features over a transaction-pooled PgBouncer." - which is why nobody uses it, because pgbouncer claims this is "broken". It's not any more broken than it is to switch out the PostgreSQL session underneath a connection that uses multiple transactions. Pgbouncer can do better here and make this clearer and more accommodating of real world database drivers.
-
aiopg vs asyncpg vs psycopg3
asyncpg: 5.5k starts, last commit recently, ~150 issues, some incompatibility, few open PRs, extensive README. Includes benchmark showing it's supposedly 3x faster than aiopg and psycopg2, psycopg3 is not mentioned in the benchmark.
-
Announcing Quart-DB
Quart-DB uses asyncpg to manage the connections and buildpg to parse the named parameter bindings.
-
Cascade of doom: JIT, and how a Postgres update led to 70% failure on a critical national service
Simple query runs long when DB schema contains thousands of tables #186
-
Large database to python - SQL advantageous? (mysql)
Highly recommend https://github.com/MagicStack/asyncpg.
-
Postgrey - Simple, Fast & Async Library for PostgreSQL
🐘 Simple, Fast, Async & ORM PostgreSQL database client based on Asyncpg for Python.
-
New major versions of Flask, Jinja, Click, and Werkzeug released!
What we get with the async part of this release - If you would like to run an async library or your own async code from a flask route you can do that now. This is super useful, where let's say we have some async code that fetches data from many sources concurrently, or call multiple a few ML prediction endpoints at the same time (as long as they don't time out) using httpx and respond with some sort of outcome, or finally try that cool new async-only database library. A current (v2) limitation is that the you can't make concurrent requests using just the current asyncio implementation (an alternative with Flask API and ASGI: Quart). Typically in production gunicorn or uwsgi + threads/processes/gevent-eventlet is used and this makes Flask behave asynchronously. More here and here if interested.
-
Migrating to SQLAlchemy 2.0
Agreed. I will never recommend an ORM, things simply spiraled out of control for medium to large-ish projects that had more than 2 developers. Even with "best practices", code ended up having a mix of raw SQL and ORM-style queries, and it was hard to reason about the code.
Since switching to asyncpg [0] these problems have vanished. It commands a deeper knowledge of actual SQL, but I would argue this knowledge is absolutely necessary and one of the disadvantages of an ORM is that it makes the SQL that is eventually run opaque.
Not sure if there are equivalents to asyncpg for other RDBMS's.
pgbouncer
-
Minha jornada de otimização de uma aplicação django
Pgbouncer - resolvia o problema do limite de conexões no postgres. Mas a API “saudável” manteve o número de conexões baixo o suficiente.
-
PgBouncer is useful, important, and fraught with peril
Pgbouncer maintainer here. Overall I think this is a great description of the tradeoffs that PgBouncer brings and how to work around/manage them. I'm actively working on fixing quite a few of the issues in this blog though
1. Named protocol-level prepared statements in transaction mode has a PR that's pretty close to being merged: https://github.com/pgbouncer/pgbouncer/pull/845
-
Supavisor: Scaling Postgres to 1 Million Connections
A common solution is connection pooling. Supabase currently offers pgbouncer which is single-threaded, making it difficult to scale. We've seen some novel ways to scale pgbouncer, but we have a few other goals in mind for our platform.
-
Citus 12: Schema-based sharding for PostgreSQL
Great observation! :)
We worked upstream to have `search_path` properly handled (tracked per client) by pgbouncer.
https://github.com/pgbouncer/pgbouncer/commit/8c18fc4d213ad4...
Check config.md in that commit for a verbose, humanized description.
-
The Database Package Manager for PostgreSQL Trusted Language Extensions
Connection poolers like PgBouncer [1] (traditional) and Supabase's Supavisor [2] (new) come to mind.
-
Show HN: Supavisor – a Postgres connection pooler written in Elixir
Interesting PR. It looks like https://github.com/pgbouncer/pgbouncer/pull/757 is more inline with multiple clients using the same prepared statements. I'll be watching that closely.
-
Reddit Recap Series: Backend Performance Tuning
Our backend services are using pgBouncer to pool PostgreSQL connections. During load testing, we found 2 problematic areas:
- Launch HN: Activepieces (YC S22) – Open-Source Zapier Alternative
-
PostgreSQL with PGBouncer
I also have a related pgbouncer PR open. Feel free to try it out, I'm planning to get it into the next pgbouncer release: https://github.com/pgbouncer/pgbouncer/pull/666
What are some alternatives?
psycopg2 - PostgreSQL database adapter for the Python programming language
aiopg - aiopg is a library for accessing a PostgreSQL database from the asyncio
odyssey - Scalable PostgreSQL connection pooler
pymssql - Official home for the pymssql source code.
awesome-mysql - A curated list of awesome MySQL software, libraries, tools and resources
mysql-python - MySQLdb is a Python DB API-2.0 compliant library to interact with MySQL 3.23-5.1 (unofficial mirror)
psycopg - New generation PostgreSQL database adapter for the Python programming language
PyMySQL - MySQL client library for Python
PyPika - PyPika is a python SQL query builder that exposes the full richness of the SQL language using a syntax that reflects the resulting query. PyPika excels at all sorts of SQL queries but is especially useful for data analysis.
pgcat - PostgreSQL pooler with sharding, load balancing and failover support. [Moved to: https://github.com/postgresml/pgcat]
TimescaleDB - An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
pgcat - PostgreSQL pooler with sharding, load balancing and failover support.