t-digest VS clojure

> I get a lot of benefit from this information but somehow it feels shallow.

I take a longer view to this. For example, a few years ago I read about an algorithm to calculate percentiles in real time. [0]

It literally just came up at work today. I haven't used that information but maybe two times since I read it, but it was super relevant today and saved my team potential weeks of development.

So maybe it's not so shallow.

But to your actual question, I have a similar problem. The best I can say is that deadlines help. I usually put down the HN and Youtube when I have a deadline coming up. And not just at work. I make sure my hobbies have deadlines too.

I tell people when I think something will be done, so they start bugging me about it when it doesn't get done, so that I have a "deadline". Also one of my hobbies is pixel light shows for holidays, which come with excellent natural deadlines -- it has to be done by the holiday or it's useless.

So either find an "accountability buddy" who will hold you to your self imposed deadlines, or find a hobby that has natural deadlines, like certain calendar dates, or annual conventions or contests that you need to be done by.

[0] https://github.com/tdunning/t-digest

I am enamored by data structures in the sketch/summary/probabilistic family: t-digest[1], q-digest[2], count-min sketch[3], matrix-sketch[4], graph-sketch[5][6], Misra-Gries sketch[7], top-k/spacesaving sketch[8], &c.

What I like about them is that they give me a set of engineering tradeoffs that I typically don't have access to: accuracy-speed[9] or accuracy-space. There have been too many times that I've had to say, "I wish I could do this, but it would take too much time/space to compute." Most of these problems still work even if the accuracy is not 100%. And furthermore, many (if not all of these) can tune accuracy to by parameter adjustment anyways. They tend to have favorable combinatorial properties ie: they form monoids or semigroups under merge operations. In short, a property of data structures that gave me the ability to solve problems I couldn't before.

I hope they are as useful or intriguing to you as they are to me.

1. https://github.com/tdunning/t-digest

2. https://pdsa.readthedocs.io/en/latest/rank/qdigest.html

3. https://florian.github.io/count-min-sketch/

4. https://www.cs.yale.edu/homes/el327/papers/simpleMatrixSketc...

5. https://www.juanlopes.net/poly18/poly18-juan-lopes.pdf

6. https://courses.engr.illinois.edu/cs498abd/fa2020/slides/20-...

7. https://people.csail.mit.edu/rrw/6.045-2017/encalgs-mg.pdf

8. https://www.sciencedirect.com/science/article/abs/pii/S00200...

9. It may better be described as error-speed and error-space, but I've avoided the term error because the term for programming audiences typically evokes the idea of logic errors and what I mean is statistical error.

Ah, I misunderstood what you meant. If you are reporting static buckets I get how that is better than what folks typically do but how do you know the buckets a priori? Others back their histograms with things like https://github.com/tdunning/t-digest. It is pretty powerful as the buckets are dynamic based on the data and histograms can be added together.

Yes, but if you need to estimate median on the fly (e.g., over a stream of data) or in parallel there are better ways.

There are some newer data structures that take this to the next level such as T-Digest[1], which remains extremely accurate even when determining percentiles at the very tail end (like 99.999%)

[1]: https://arxiv.org/pdf/1902.04023.pdf / https://github.com/tdunning/t-digest

[2] https://github.com/tdunning/t-digest

T-Digest

This is pretty cool. The title would be a bit more descriptive if it were “Fast Rolling Quantile Filters for Python”, since the high-pass/low-pass filter functionality seems to be the focus.

The README mentions it uses binary heaps - if you’re willing to accept some (bounded) approximation, then it should be possible to reduce memory usage and somewhat reduce runtime by using a sketching data structure like Dunning’s t-digest: https://github.com/tdunning/t-digest/blob/main/docs/t-digest....

There is an open source Python implementation, although I haven’t used it and can’t vouch for its quality: https://github.com/CamDavidsonPilon/tdigest

5. Clojure - $96,381

I have a tangential question that is related to this cool new feature.

Warning: the question I ask comes from a part of my brain that is currently melted due to heavy thinking.

Context: I write a fair amount of Clojure, and in Lisps the code itself is a tree. Just like this F# parallel graph type-checker. In Lisps, one would use Macros to perform compile-time computation to accomplish something like this, I think.

More context: Idris2 allows for first class type-driven development, where the types are passed around and used to formally specify program behavior, even down to the value of a particular definition.

Given that this F# feature enables parallel analysis, wouldn't it make sense to do all of our development in a Lisp-like Trie structure where the types are simply part of the program itself, like in Idris2?

Also related, is this similar to how HVM works with their "Interaction nets"?

https://github.com/HigherOrderCO/HVM

https://www.idris-lang.org/

https://clojure.org/

I'm afraid I don't even understand what the difference between code, data, and types are anymore... it used to make sense, but these new languages have dissolved those boundaries in my mind, and I am not sure how to build it back up again.

As an analogy - my face hasn't changed all that much in a past few years, and I haven't changed my profile picture in those few years. Does it really mean that I'm unmaintained/dead?

> Where can I find latest documentation [...]?

The answer is still https://clojure.org/. And https://clojuredocs.org/ but it's community-maintained so might occasionally be missing some things right after they're released. E.g. as of this moment Clojure 1.11 is still not there since the maintainer of the website has some technical issues deploying the updated version of the website.

For me personally, the best API-level documentation is the source code.

> Where can I find [...] tools / libraries in a easy to use page or section?

There's no central repository of all the available things since they can be loaded from many places (Clojars, Maven Central, other Maven repositories, S3, Git, local files).

But there are community-maintained lists, like the one you've mentioned at https://www.clojure-toolbox.com (fully manual, AFAIK) or the one at https://phronmophobic.github.io/dewey/search.html (automated but only for GitHub). Perhaps there are others but I'm not familiar with them - most of the time, I myself don't find that much value in such services as I'm usually able to find things with a regular web search engine or ask the community when I need something in particular.

They are written in Java, and implement a bunch of interfaces, so the implementation looks complicated, but they are basically just classes with head and tail fields.

https://github.com/clojure/clojure/blob/master/src/jvm/cloju...

Like this.

As a Java/Scala user you should check out Clojure! It is highly recommended (https://clojure.org)

Lisp is not a programming language, but a family of languages ​​with many dialects. The most famous dialects include Common Lisp, Clojure, Scheme and Racket. So after deciding that I was going to learn Lisp, I had to choose one of its dialects.

I learned Clojure to implement a plugin for Metabase (the tool my former company used for creating business dashboards). I probably won’t ever use the language anymore in the future, but learning functional programming was fun and eye-opening.

I thought you might be trolling. But then when I looked at the Clojure repo on Github https://github.com/clojure/clojure the last commit was 2 months back. There is some merit in your arguments.