clojure VS minisketch

5. Clojure - $96,381

I have a tangential question that is related to this cool new feature.

Warning: the question I ask comes from a part of my brain that is currently melted due to heavy thinking.

Context: I write a fair amount of Clojure, and in Lisps the code itself is a tree. Just like this F# parallel graph type-checker. In Lisps, one would use Macros to perform compile-time computation to accomplish something like this, I think.

More context: Idris2 allows for first class type-driven development, where the types are passed around and used to formally specify program behavior, even down to the value of a particular definition.

Given that this F# feature enables parallel analysis, wouldn't it make sense to do all of our development in a Lisp-like Trie structure where the types are simply part of the program itself, like in Idris2?

Also related, is this similar to how HVM works with their "Interaction nets"?

https://github.com/HigherOrderCO/HVM

https://www.idris-lang.org/

https://clojure.org/

I'm afraid I don't even understand what the difference between code, data, and types are anymore... it used to make sense, but these new languages have dissolved those boundaries in my mind, and I am not sure how to build it back up again.

As an analogy - my face hasn't changed all that much in a past few years, and I haven't changed my profile picture in those few years. Does it really mean that I'm unmaintained/dead?

> Where can I find latest documentation [...]?

The answer is still https://clojure.org/. And https://clojuredocs.org/ but it's community-maintained so might occasionally be missing some things right after they're released. E.g. as of this moment Clojure 1.11 is still not there since the maintainer of the website has some technical issues deploying the updated version of the website.

For me personally, the best API-level documentation is the source code.

> Where can I find [...] tools / libraries in a easy to use page or section?

There's no central repository of all the available things since they can be loaded from many places (Clojars, Maven Central, other Maven repositories, S3, Git, local files).

But there are community-maintained lists, like the one you've mentioned at https://www.clojure-toolbox.com (fully manual, AFAIK) or the one at https://phronmophobic.github.io/dewey/search.html (automated but only for GitHub). Perhaps there are others but I'm not familiar with them - most of the time, I myself don't find that much value in such services as I'm usually able to find things with a regular web search engine or ask the community when I need something in particular.

They are written in Java, and implement a bunch of interfaces, so the implementation looks complicated, but they are basically just classes with head and tail fields.

https://github.com/clojure/clojure/blob/master/src/jvm/cloju...

As a Java/Scala user you should check out Clojure! It is highly recommended (https://clojure.org)

Lisp is not a programming language, but a family of languages ​​with many dialects. The most famous dialects include Common Lisp, Clojure, Scheme and Racket. So after deciding that I was going to learn Lisp, I had to choose one of its dialects.

I thought you might be trolling. But then when I looked at the Clojure repo on Github https://github.com/clojure/clojure the last commit was 2 months back. There is some merit in your arguments.

1 - partial is defining a new function that ignores the type hints from func, and would introduce boxing. It also can introduce a performance hit for (remaining) argument arities > 3, since it automatically invokes a varargs variant instead of providing a concrete arity. With the varargs version, in profiling you may see RestFn showing up on hot paths, which is the varargs implementation having to munge seqs every invocation instead of being able to use concrete args matching discrete arities. Depending on the frequency of invocation this may impact performance.

I love Clojure, but the Java source is oddly formatted which I never understood: https://github.com/clojure/clojure/blob/527b330045ef35b47a96...

I'm using Leiningen to set up the project, so it seems that the guide on clojure.org does not apply. And in the Leiningen Docs I couldn't find anything...

Since the protocol appears to use adhoc synchronization, the authors might be interested in https://github.com/sipa/minisketch/ which is a library that implements a data structure (pinsketch) that allows two parties to synchronize their sets of m b-bit elements which differ by c entries using only b*c bits. A naive protocol would use m*b bits instead, which is potentially much larger.

I'd guess that under normal usage the message densities probably don't justify such efficient means-- we developed this library for use in bitcoin targeting rates on the order of a dozen new messages per second and where every participant has many peers with potentially differing sets--, but it's still probably worth being aware of. The pinsketch is always equal or more efficient than a naive approach, but may not be worth the complexity.

The somewhat better known IBLT data structure has constant overheads that make it less efficient than even naive synchronization until the set differences are fairly large (particular when the element hashes are small); so some applications that evaluated and eschewed IBLT might find pinsketch applicable.

Here is one not on the list so far:

Set Sketches. They allow you compute the difference between two sets (for example to see if data has been replicated between two nodes) WITHOUT transmitting all the keys in one set to another.

Say you have two sets of the numbers [1, ..., 1million] all 32 bit integers, and you know one set is missing 2 random numbers. Set sketches allow you to send a "set checksum" that is only 64 BITS which allows the other party to compute those missing numbers. A naive algorithm would require 1MB of data be transferred to calculate the same thing.

*(in particular pin sketch https://github.com/sipa/minisketch).

How about a pinsketch:

A pinsketch is a set that takes a specified amount of memory and into which you can insert and remove set members or even add whole sets in time O(memory size). You can insert an unbounded number of entries, and at any time that it has equal or fewer entries than the size you can decode the list of members.

For an example usage, say I have a list of ten million IP addresses of people who have DOS attacked my systems recently. I want to send my list to you over an expensive iridium connection, so I don't want to just send the 40MiB list. Fortunately you've been making your own observations (and maybe have stale data from me), and we don't expect our lists to differ by more than 1000 entries. So I make and maintain a pinsketch with size 1000 which takes 4000 bytes (1000 * 4bytes because IP addresses are 32-bits). Then to send you an update I just send it over. You maintain your own pinsketch of addresses, you subtract it from the one I sent and then you decode it. If the number of entries different between us is under 1000 you're guaranteed to learn the difference (otherwise the decode will fail, or give a false decode with odds ~= 1/2^(1000)).

Bonus: We don't need to know in advance how different our sets are-- I can send the sketch in units as small as one word at a time (32-bits in this case) and stop sending once you've got enough to decode.

Here is an implementation I contributed to: https://github.com/sipa/minisketch/

There is a application related data-structure called an inverted bloom lookup table (IBLT) that accomplishes the same task. Its encoding and especially decoding is much faster, and it has asymptotically the same communications efficiency. However, the constant factors on the communications efficiency are poor so for 'small' in set difference (like the 1000 above) it has a rather high overhead factor, and it can't guarantee decoding. I think that makes it much less magical, though it may be the right tool for some applications.

IBLT also has the benefit that it the decoder is a fun bit of code golf to implement.

I love the set reconciliation structures like the IBLT (Iterative Bloom Lookup Table) and BCH set digests like minisketch.

https://github.com/sipa/minisketch

Lets say you have a set of a billion items. Someone else has mostly the same set but they differ by 10 items. These let you exchange messages that would fit in one UDP packet to reconcile the sets.

Sipa is working on a better version of that for a while. The technical term is a "set reconciliation protocol", but Bitcoin Core been doing a more basic version of this for a while. Note that the "BCH" there isn't the same as Bcash

I'd also have to say that Zig is a pretty neat library for this. In order to implement PBS I needed the MiniSketch-library (written in C/C++) and I'll have to say that integrating with it has been a breeze. Some fiddling in build.zig so that I can avoid Makefile, and after that everything has worked amazingly.

Networks that need to constrain themselves to limited typologies to avoid traffic magnification do so at the expense of robustness, especially against active attackers that grind their identifiers to gain privileged positions.

Maybe this is a space where efficient reconciliation ( https://github.com/sipa/minisketch/ ) could help-- certainly if the goal were to flood messages to participants reconciliation can give almost optimal communication without compromising robustness.