baseplate.py
pgbouncer
baseplate.py | pgbouncer | |
---|---|---|
11 | 34 | |
530 | 2,692 | |
0.2% | 3.1% | |
8.5 | 8.7 | |
4 days ago | 8 days ago | |
Python | C | |
BSD 3-clause "New" or "Revised" License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
baseplate.py
-
The .zip TLD sucks and it needs to be immediately revoked.
Almost any download link on the internet (for example an attachment to any Wordpress blog post) could serve as an example. Since we are on a programming subreddit let's use Github as an example. When you open a repository and click the "Download ZIP" button it's a link straight to an URL like this one: https://github.com/reddit/baseplate.py/archive/refs/heads/develop.zip (this particular one is for Reddit's Python library).
-
Python use by SWEs
Even Reddit has python backends https://github.com/reddit/baseplate.py based on Pyramid. They also have a go one. https://github.com/reddit/baseplate.go
-
Reddit System Design/Architecture
there's a multitude of services in reddit's architecture. as far as i can tell, they mostly using reddit's baseplate framework (which has implementations in both python and go).
-
Reddit Recap Series: Backend Performance Tuning
Finally, the problem that we didn’t experience directly, but it was mentioned during consultations with another team that had experience with pgBouncer: the Baseplate.py framework that both of us are using sometimes leaked the connections, leaving them open after the request, but not returning them back into the pool.
- What is an example of a fully finished python software product on github?
-
Is the Pyramid framework dead?
Also reddit team using pyramid for services https://github.com/reddit/baseplate.py
-
is flask only for "smaller" projects or it can also be used for large scalable projects ?
Reddit is built on neither Flask nor Django. The old monolith predates Flask and Django and is built on its own framework. Our new microservices are built on our Baseplate.py framework.
-
Evolving Reddit’s ML Model Deployment and Serving Architecture
Minsky is an internal baseplate.py (Reddit’s python web services framework) thrift service owned by Reddit’s Machine Learning team that serves data or derivations of data related to content relevance heuristics — such as similarity between subreddits, a subreddits topic or a users propensity for a given subreddit — from various data stores such as Cassandra or in process caches. Clients of Minsky use this data to improve Redditor’s experiences with the most relevant content. Over the last few years a set of new ML capabilities, referred to as Gazette, were built into Minsky. Gazette is responsible for serving ML model inferences for personalization tasks along with configuration based schema resolution and feature fetching / transformation.
-
Deadline Budget Propagation for Baseplate.py
Baseplate is implemented in Python and Go, and although they share the same main functionality, smaller features differ between the two. One such feature that was previously on the Go implementation but not Python was deadline budget propagation, which passes on the remaining timeout available from the initial client request all the way through the server and any other requests that may follow. The lack of this feature in Baseplate.py meant that many resources were being wasted by servers doing unnecessary work, despite clients no longer awaiting their response due to timeout.
-
Solving The Three Stooges Problem
In order to make this solution work, you’ll need a web stack that can handle many concurrent requests. Reddit’s stack for most microservices is Python 3, Baseplate, and gevent. Django/Flask also work well when run with gevent. gevent is a Python library that transparently enables your microservice to handle high concurrency and I/O without requiring changes to your code. It is the secret sauce that allows you to run tens of thousands of pseudo-threads called greenlets (one per concurrent request) on a small number of instances. It allows for threads handling concurrent duplicate requests to be enqueued while waiting to acquire the lock, and then for those queues to be drained as threads acquire the lock and execute serially, all without exhausting the thread pool.
pgbouncer
-
MongoDB and Load Balancer Support
Thanks to MongoDB drivers all consistently providing connection monitoring and pooling functionality, external connection pooling solutions aren't required (ex: Pgpool, PgBouncer). This allows applications built using MongoDB drivers to be resilient and scalable out of the box, but based on what we understand regarding the number of connections applications establish to MongoDB clusters it stands to reason that at a certain point as our application deployments increase, so will our connections.
-
Minha jornada de otimização de uma aplicação django
Pgbouncer - resolvia o problema do limite de conexões no postgres. Mas a API “saudável” manteve o número de conexões baixo o suficiente.
- PgBouncer 1.21.0 – "The one with prepared statements"
- Pgbouncer adds support for prepared statements
-
PgBouncer is useful, important, and fraught with peril
Pgbouncer maintainer here. Overall I think this is a great description of the tradeoffs that PgBouncer brings and how to work around/manage them. I'm actively working on fixing quite a few of the issues in this blog though
1. Named protocol-level prepared statements in transaction mode has a PR that's pretty close to being merged: https://github.com/pgbouncer/pgbouncer/pull/845
-
Supavisor: Scaling Postgres to 1 Million Connections
A common solution is connection pooling. Supabase currently offers pgbouncer which is single-threaded, making it difficult to scale. We've seen some novel ways to scale pgbouncer, but we have a few other goals in mind for our platform.
-
Citus 12: Schema-based sharding for PostgreSQL
Great observation! :)
We worked upstream to have `search_path` properly handled (tracked per client) by pgbouncer.
https://github.com/pgbouncer/pgbouncer/commit/8c18fc4d213ad4...
Check config.md in that commit for a verbose, humanized description.
What are some alternatives?
xhtml2pdf - A library for converting HTML into PDFs using ReportLab
odyssey - Scalable PostgreSQL connection pooler
Apache Thrift - Apache Thrift
asyncpg - A fast PostgreSQL Database Client Library for Python/asyncio.
Pyramid - Pyramid - A Python web framework
pgcat - PostgreSQL pooler with sharding, load balancing and failover support. [Moved to: https://github.com/postgresml/pgcat]
rtv - Browse Reddit from your terminal
TimescaleDB - An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
pottery - Redis for humans. 🌎🌍🌏
pgcat - PostgreSQL pooler with sharding, load balancing and failover support.
redsync - Distributed mutual exclusion lock using Redis for Go
rds-auth-proxy - A "passwordless" login experience for your AWS RDS