pgBackRest
Disruptor
pgBackRest | Disruptor | |
---|---|---|
13 | 30 | |
2,209 | 17,029 | |
2.6% | 0.4% | |
9.2 | 5.4 | |
6 days ago | 4 months ago | |
C | Java | |
GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pgBackRest
-
pgBackRest: PostgreSQL S3 backups
This tutorial explains how to backup PostgreSQL database using pgBackRest and S3.
-
Anything can be a message queue if you use it wrongly enough
This isn't theoretical; many companies do PostgreSQL async 1:N physical replication, by using e.g. https://pgbackrest.org/ to have the primary push WAL segment files (a.k.a. "the last n milliseconds of packets" in the write-ahead log) as objects to S3, and then to have all read-replicas fetch from S3 and replay.
> You could do even better if you out-of-band signal the readiness so you do not need to poll while idle.
S3 and its clones have "object lifecycle notifications", where you can be informed by a push-based mechanism whenever a new object is put into the bucket.
But — what do you have to do, to get these notifications?
Subscribe to a message queue that S3 puts them into.
-
Kubernetes postgres backups
I haven't explored the territory in awhile but for bare-metal, you can't go wrong with Percona Distribution, which includes pgBackRest and a minimal web-ui. No one ever got fired for using Percona, etc.
- pgBackRest - Reliable PostgreSQL Backup & Restore
- pgBackRest - have you used it and what was your experience?
-
How to backup database
Check out pgBackRest
-
Use One Big Server
I found this approach pretty cool in that regard: https://github.com/pgbackrest/pgbackrest
- Moving from Oracle to Postgres, what should I know?
-
How do you back up your databases?
Something like PG barman or pg backrest could be good for you on the Postgres side.
-
Cloud SQL is not great
Backups are limited. These days, pgbackrest is the go-to backup solution for PostgreSQL, and having used it I am very impressed so far. It provides full backups, differential, and incremental, as well as archiving of WAL segments for point in time recovery. It allows great flexibility in schedules and destinations for backups, how long to keep backups for, how many full backups. For example, you can have backups made to a local disk, and other backups to an external S3-compatible bucket, each with their own settings and schedules (e.g., scheduled via cron).
Disruptor
-
Gnet is the fastest networking framework in Go
https://lmax-exchange.github.io/disruptor/#_what_is_the_disr.... Unfortunately IIUC writing this in Go still prevents the spin-locked acceptor thread from achieving the kind of performance you could get in a non-GC language, unless you chose to disable GC, so I'd guess Envoy is still faster.
https://gnet.host/docs/quickstart/ it's nice that you can use this simply though. Envoy is kind of tricky to setup with custom filters, so most of the time it's just a standalone binary.
[0] https://blog.envoyproxy.io/envoy-threading-model-a8d44b92231...
[1] https://lmax-exchange.github.io/disruptor/#_what_is_the_disr...
-
A lock-free ring-buffer with contiguous reservations (2019)
See also the Java LMAX Disruptor https://github.com/LMAX-Exchange/disruptor
I've built a similar lock-free ring buffer in C++11 https://github.com/posterior/loom/blob/master/doc/adapting.m...
-
JEP Draft: Deprecate Memory-Access Methods in Sun.misc.Unsafe for Removal
"Why we chose Java for our High-Frequency Trading application"
https://medium.com/@jadsarmo/why-we-chose-java-for-our-high-...
LMAX Disruptor customers
https://lmax-exchange.github.io/disruptor/
Among many other examples.
-
LMAX Disruptor – High Performance Inter-Thread Messaging Library
Current documentation
https://lmax-exchange.github.io/disruptor/
-
Progress on No-GIL CPython
LMAX Disruptor has on their wiki that average latency to send a message from one thread to another at 53 nanoseconds. For comparison a mutex is like 25 nanoseconds and more if Contended but a mutex is point to point synchronization.
The great thing about it is that multiple threads can receive the same message without much more effort.
https://github.com/LMAX-Exchange/disruptor/wiki/Performance-...
https://gist.github.com/rmacy/2879257
I am dreaming of language that is similar to Smalltalk that stays single threaded until it makes sense to parallise.
I am looking for problems to parallelism that are not big data. Parallelism is like adding more cars to the road rather than increasing the speed of the car. But what does a desktop or mobile user need to do locally that could take advantage of the mathematical power of a computer? I'm still searching.
- Disruptor 4.0.0 Released
-
Anything can be a message queue if you use it wrongly enough
Database config should be two connection strings, 1 for the admin user that creates the tables and anther for the queue user. Everything else should be stored in the database itself. Each queue should be in its own set of tables. Large blobs may or may not be referenced to an external file.
Shouldn't a message send be worst case a CAS. It really seems like all the work around garbage collection would have some use for in-memory high speed queues.
Are you familiar with the LMAX Disruptor? Is is a Java based cross thread messaging library used for day trading applications.
https://lmax-exchange.github.io/disruptor/
-
Any library you would like to recommend to others as it helps you a lot? For me, mapstruct is one of them. Hopefully I would hear some other nice libraries I never try.
Disruptor for inter-thread messaging
-
Measuring how much Rust's bounds checking actually costs
I have never worked in any industries where a perf margin was that small. It is funny, in HFT there are folks using Lmax (Java) and then you have folks writing their own TCP/IP stacks on FPGAs to do trading.
What are some alternatives?
Barman - Barman - Backup and Recovery Manager for PostgreSQL
JCTools
wal-g - Archival and Restoration for databases in the Cloud
Agrona - High Performance data structures and utility methods for Java
docker-postgres-wale - Postgres docker container with WALE-E installed
fastutil - fastutil extends the Java™ Collections Framework by providing type-specific maps, sets, lists and queues.
wal-e - Continuous Archiving for Postgres
MPMCQueue.h - A bounded multi-producer multi-consumer concurrent queue written in C++11
pghoard - PostgreSQL® backup and restore service
Eclipse Collections - Eclipse Collections is a collections framework for Java with optimized data structures and a rich, functional and fluent API.
postgres - Docker Official Image packaging for Postgres
Javolution