Our great sponsors
-
We should consider the following properties when evaluating ID formats and generation algorithms:
1. Private: you shouldn’t be able to gain information about the system using the IDs from an ID alone. E.g. document enumeration attacks like what happened with Parker (https://www.wired.com/story/parler-hack-data-public-posts-im...)
2. B-tree/cache friendly: newly created IDs should all exist in a narrow range of values. This is helpful for databases.
3. Stateless: ideally you shouldn’t need to know the current state of the system to create a new ID.
4. Human-friendly: IDs should be easily dictated, copied, pasted, etc. This means they should be encodable as text that is short and does not include ambiguous characters.
Some of the these properties are in conflict. Statelessness is achieved by randomly generating long IDs, but people don’t like reading or typing long IDs.
Different use cases will need these properties in varying amounts. If you don’t intend to expose the IDs to users, (4) doesn’t matter. Just use long, randomly generated byte strings prepended with the date. Most databases have a UUID type that fits the bill.
If users are going to be working with IDs, that’s more complicated. If not every document has a user-facing ID, just go with the non-user-facing ID like before, and generate a shorter, random, stateful ID as needed.
I don’t think NanoID prepends the date, which means it won’t be efficient when inserting large numbers of IDs into a large index. They also default to using ambiguous characters like 1 and I and l. Also no error code. But they are shorter than UUIDs. So it doesn’t meet property (2), and it only kind of meets property (4). NanoIDs are random, so you’re probably safe from enumeration attacks (1). NanoIDs mostly leave Statelessness as a decision for the user. They have a nice tool that helps estimate how long the IDs should be (https://zelark.github.io/nano-id-cc/) for a given collision resistance.
I think we can do better overall. Bitcoin uses a good encoding scheme called base58check. It generates fairly short IDs and uses an error code at the end. I think it could be refined for non-bitcoin purposes, but it’s already pretty good.
A 128-bit value like the ASCII string “hackernewstestid” is encoded as “Dtajqjz5pptWcmGrNcwBx7”. It’s about 2/3 the size of the equivalent UUID, even with the (unnecessarily long for this use case) error checking code.
I’d like to see a small ID standard that meets the above requirements and has a choice for either stateless and long or stateful and short. Maybe another choice for secure random or insecure. But all options would have binary form and a text form. The text form would use something similar to base58check, but probably with a smaller (or user-determined) length for the error code.
-
I wonder how this might compare to just storing regular autoincrementing ints in the database, and converting to/from hashids (https://hashids.org/) at the edge. It eliminates the collision concern and stores more compactly at the cost of a tiny amount of encode/decode when processing requests. You’d want to push it down as close to the database layer as possible to avoid inadvertent int ID leaks; I added native hashids support to clickhouse but I’m not sure what other database support might entail.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
uuid-base58
Generate a RFC4122 compliant v4 UUID and return it encoded in base-58. This is great for creating unique IDs which only consume 22 characters of storage. Also provides base-58 encoding and decoding.
looks like there's an impl already: https://github.com/cbschuld/uuid-base58
-
ULID hits most of these, and can be converted to UUID for use with databases supporting this datatype (not a strong column): https://github.com/ulid/spec
Related posts
- Adding type safety to object IDs in TypeScript
- You Don't Need UUID
- I created a CLI tool that writes unit tests with GPT-4 (with one command, I created tests for Lodash repo with 90% code coverage and found 13 bugs)
- Analyzing New Unique Identifier Formats (UUIDv6, UUIDv7, and UUIDv8) (2022)
- Feedback to Writing Processor for Creative Writing