suggest VS ordiri

You are welcome. Thanks are too rarely offered. :-)

You may also be interested in word stemming ( such as used by snowball stemmer in https://github.com/c-blake/nimsearch ) or other NLP techniques, but I don't know how internationalized/multi-lingual that stuff is, but conceptually you might want "series of stemmed words" to be the content fragments of interest.

Similarity scores have many applications. Weights on graph of cancelled downloads ranked by size might be one. :)

Of course, for your specific "truncation" problem, you might also be able to just do an edit distance against the much smaller filenames and compare data prefixes in files or use a SHA256 of a content-based first slice. ( There are edit distance algos in Nim in https://github.com/c-blake/cligen/blob/master/cligen/textUt.... as well as in https://github.com/c-blake/suggest ).

Or, you could do a little program like ndup/sh/ndup to create a "mirrored file tree" of such content-based slices then you could use any true duplicate-file finder (like https://github.com/c-blake/bu/blob/main/dups.nim) on the little signature system to identify duplicates and go from path suffixes in those clusters back to the main filesystem. Of course, a single KV store within one or two files would be more efficient than thousands of tiny files. There are many possibilities.

As jamra correctly points out, the entry point to this (which gets a lot of traction on HN) is indeed attacking a strawman tutorial-written-on-an-airplane algorithm. So, the 1M speed-up is majorly over-hyped.

That said, the technique is not wholly without merit, but does carry certain "risk-reward" trade offs related to latency in the memory/storage system because of SymSpell's reliance upon large hash tables. For details see https://github.com/c-blake/suggest

I self host a huge amount of stuff on-top of a custom cloud platform I built using a Kubernetes cluster deployed as a tenant of my cloud.

I run a few servers which all have an "ordlet" installed, akin to the kubelet, which configure network namespaces for isolated tenant networking and boots virtual machines which use a EC2 style metadata server to fetch their boot script, which for this purpose configures a HA kubernetes cluster that then uses ArgoCD to fetch all the manifests from my git repo using an AppSet.

It's so incredibly over complicated and over engineered, it's a lot of fun :)

https://github.com/ordiri/ordiri

https://github.com/ordiri/ordiri

I've been working on an incredibly over engineered platform to run my home lab and to help learn about some lower levels of the stack I don't get to play with much in my day job as a consultant.

It's based on the kubernetes api server so while it doesn't understand "pods" or even proper "namespaces", it uses the same YAML resource model and api server code.