postgres-elasticsearch-fdw
ksuid
Our great sponsors
postgres-elasticsearch-fdw | ksuid | |
---|---|---|
3 | 38 | |
106 | 4,682 | |
- | 2.2% | |
4.2 | 3.1 | |
29 days ago | 7 months ago | |
Python | Go | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
postgres-elasticsearch-fdw
- Full-text search engine with PostgreSQL (part 2): Postgres vs. Elasticsearch
-
Postgres Full Text Search vs. the Rest
My experience with Postgres FTS (did a comparison with Elastic a couple years back), is that filtering works fine and is speedy enough, but ranking crumbles when the resulting set is large.
If you have a large-ish data set with lots of similar data (4M addresses and location names was the test case), Postgres FTS just doesn't perform.
There is no index that helps scoring results. You would have to install an extension like RUM index (https://github.com/postgrespro/rum) to improve this, which may or may not be an option (often not if you use managed databases).
If you want a best of both worlds, one could investigate this extensions (again, often not an option for managed databases): https://github.com/matthewfranglen/postgres-elasticsearch-fd...
Either way, writing something that indexes your postgres database into elastic/opensearch is a one time investment that usually pays off in the long run.
-
Lesser Known PostgreSQL Features
I used a foreign data wrapper to query elasticsearch indexes from within postgres.[0]
It pushed alot of complexity down away from higher-level app developers not familiar with ES patterns.
[0]: https://github.com/matthewfranglen/postgres-elasticsearch-fd...
ksuid
- What happens after 100 years?
-
Zero Downtime Postgres Upgrades
OP here - we avoid sequences in all but one part of our application due to a dependency. We use [KSUIDs][1] and UUID v4 in various places. This one "gotcha" applies to any sequence, so it's worth calling out as general advice when running a migration like this.
[1]: https://segment.com/blog/a-brief-history-of-the-uuid/
-
Bye Sequence, Hello UUIDv7
UUID v4 isn't large enough to prevent collisions, that is why segment.io created https://github.com/segmentio/ksuid which is 160bit vs the 128bit of a UUIDv4.
- You Don't Need UUID
- A Brief History of the UUID
-
Type-safe, K-sortable, globally unique identifier inspired by Stripe IDs
Assuming you don't need to use UUIDv7 (or any UUID's) then https://github.com/segmentio/ksuid provides a much bigger keyspace. You could just append a string prefix if you wanted to namespace, but the chance of collisions of a KSUID is many times smaller than a UUID of any version.
-
Unexpected downsides of UUID keys in PostgreSQL
KSUID's are have temporal-lexicographical order plus 128 bits of entropy, which is more than UUIDv4.
https://github.com/segmentio/ksuid
-
UUIDs are so much better than autoincrementing ids and it's not even close
That's why you use ksuid (https://segment.com/blog/a-brief-history-of-the-uuid/) or, if you're willing to go with a draft spec you could go with the new UUID formats https://www.ietf.org/archive/id/draft-ietf-uuidrev-rfc4122bi...
-
What Happened to UUIDv2?
Interesting in more history of UUIDs? Twilio Segment's blog has an amazing history lesson about how they came to be.
-
Which UUID package do you use? and why?
I use the ksuid from segment. https://github.com/segmentio/ksuid
What are some alternatives?
rum - RUM access method - inverted index with additional information in posting lists
ulid - Universally Unique Lexicographically Sortable Identifier (ULID) in Python 3
tbls - tbls is a CI-Friendly tool for document a database, written in Go.
pg-ulid - ULID Functions for PostgreSQL
nanoid - A tiny (124 bytes), secure, URL-friendly, unique string ID generator for JavaScript
pgvector - Open-source vector similarity search for Postgres
ulid-mssql - Implementation of ULID generator For Microsoft SQL Server
postgres-elasticsearch-fd
python-ksuid - A pure-Python KSUID implementation
js-id - ID generation for JavaScript & TypeScript Applications
uuid7 - UUID version 7, which are time-sortable (following the Peabody RFC4122 draft)