hamt VS PSI

This isn't a very good explanation. The wikipedia article isn't great either. I like this description:

https://github.com/mkirchner/hamt#persistent-hash-array-mapp...

The name does tell you quite a bit about what these are:

* Hash - rather than directly using the keys to navigate the structure, the keys are hashed, and the hashes are used for navigation. This turns potentially long, poorly-distributed keys into short, well-distributed keys. However, that does mean you have to compute a hash on every access, and have to deal with hash collisions. The mkirchner implementation above calls collisions "hash exhaustion", and deals with them using some generational hashing scheme. I think i'd fall back to collision lists until that was conclusively proven to be too slow.

* Trie - the tree is navigated by indexing nodes using chunks of the (hash of the) key, rather than comparing the keys in the node

* Array mapped - sparse nodes are compressed, using a bitmap to indicate which logical slots are occupied, and then only storing those. The bitmaps live in the parent node, rather than the node itself, i think? Presumably helps with fetching.

A HAMT contains a lot of small nodes. If every entry is a bitmap plus a pointer, then it's two words, and if we use five-bit chunks, then each node can be up to 32 entries, but i would imagine the majority are small, so a typical node might be 64 bytes. I worry that doing a malloc for each one would end up with a lot of overhead. Are HAMTs often implemented with some more custom memory management? Can you allocate a big block and then carve it up?

Could you do a slightly relaxed HAMT where nodes are not always fully compact, but sized to the smallest suitable power of two entries? That might let you use some sort of buddy allocation scheme. It would also let you insert and delete without having to reallocate the node. Although i suppose you can already do that by mapping a few empty slots.

There are other techniques that aren't generally included in the "Zero Knowledge Proofs" set of techniques that are perhaps more practical for general development.

For example, I fine private set intersection[1] as implemented by OpenMined a really useful primative a bunch of privacy enhancing applications can be built on top of.

My colleagues and I recently published a pre-print[2] showing how to use this for sharing locations you and another person have had in common, without being able to see other locations. The paper talks about a social network built around this but I also think there are useful applications in things like real-world games (scavenger hunts etc)

[1] https://github.com/OpenMined/PSI/blob/master/private_set_int...

[2] https://arxiv.org/abs/2210.01927

I came here to say Golomb compressed sets except now I see it's part of the question!

They are used by default in the OpenMined implementation of Private Set Intersection[1] - a multi-party computation technique.

[1] https://github.com/OpenMined/PSI/blob/master/private_set_int...

I was using OpenMinded/PSI exploring some PSI implementations, but I would like a way for the server to know the intersection size. Say Signal wants to calculate the average number of users from one person's address book (or whatever).