entt VS t-digest

EnTT is a popular alternative to flecs for C++, which has different performance/memory characteristics.

https://pastebin.com/VPypiitk This is a very small experiment i did to learn the metaprogramming features. its an ECS library using the same model as entt (https://github.com/skypjack/entt). In 200 lines or so it does the equivalent of a few thousand lines of template heavy Cpp while compiling instantly and generating good debug code.

Some walkthrough:

Line 8 declares a SparseSet type as a fairly typical template. its just a struct with arrays of type T inside. Next lines implement getters/setters for this data structure

Line 46 Base_Registry things get interesting. This is a struct that holds a bunch of SparseSet of different types, and providers getters/setters for them by type. It uses code generation to do this. The initial #insert at the start of the class injects codegen that creates structure members from the type list the struct gets on its declaration. Note also how type-lists are a native structure in the lang, no need for variadics.

Line 99 i decide to do variadic style tail templates anyway for fun. I implement a function that takes a typelist and returns the tail, and the struct is created through recursion as one would do in cpp. Getters and setters for the View struct are also implemented through recursion

Line 143 has the for expansion. This is how you overload the for loop functionality to create custom iterators.

The rest of the code is just some basic test code that runs the thing.

Since we wanted a common game simulation that would be on both the server and the client we looked into a few libraries that would fit our ECS needs. It was decided we were going to write this common part of our game in C++, but rust was considered. C++ was a familiar language for us so naturally EnTT and flecs came up right away. I had used EnTT before, writing some small demo projects, so our choice was made based on familiarity. In order to integrate with Unity we created a small C interface to communicate between our simulation code and Unity’s C#. Here’s close to what it looked like. I removed some parts for brevity sake.

Are you sure you don't want to use a C++ package manager? Libtcod is on Vcpkg and with that setup you could add the fmt library or EnTT. fmt fixes C++'s string handling and EnTT fixes everything wrong with the entities of the previous tutorials.

There's also a performance question. While we can now use Blueprint nativization to convert Blueprints to C++ the result will be a fairly naive version, fast enough for most purposes but not if you're trying to push every bit of performance. This is where you're looking at making sure you're hitting things such as using the CPU cache as well as possible for an ECS system (Look at ENTT or Flecs if you want to see what they're about and why you'd want one), or a system needing to process massive amounts of data quickly such as the Voxel Plugin.

For a modern engine you’re probably best looking at Unity’s DOTS. You may also want to check out some of the different open source ECS libraries such as flecs and EnTT are two popular ones for C++, but there’s lots of them. Largely you’ll see lots of different approaches taken, all with their own pros and cons. Not all of them will be performant (some focus more on the design benefits) while others will be optimised for certain use cases. What you should prioritise will depend on your specific needs.

You can think of DynaMix as combining one of these libraries with an ECS like entt(https://github.com/skypjack/entt)

> I get a lot of benefit from this information but somehow it feels shallow.

I take a longer view to this. For example, a few years ago I read about an algorithm to calculate percentiles in real time. [0]

It literally just came up at work today. I haven't used that information but maybe two times since I read it, but it was super relevant today and saved my team potential weeks of development.

So maybe it's not so shallow.

But to your actual question, I have a similar problem. The best I can say is that deadlines help. I usually put down the HN and Youtube when I have a deadline coming up. And not just at work. I make sure my hobbies have deadlines too.

I tell people when I think something will be done, so they start bugging me about it when it doesn't get done, so that I have a "deadline". Also one of my hobbies is pixel light shows for holidays, which come with excellent natural deadlines -- it has to be done by the holiday or it's useless.

So either find an "accountability buddy" who will hold you to your self imposed deadlines, or find a hobby that has natural deadlines, like certain calendar dates, or annual conventions or contests that you need to be done by.

[0] https://github.com/tdunning/t-digest

I am enamored by data structures in the sketch/summary/probabilistic family: t-digest[1], q-digest[2], count-min sketch[3], matrix-sketch[4], graph-sketch[5][6], Misra-Gries sketch[7], top-k/spacesaving sketch[8], &c.

What I like about them is that they give me a set of engineering tradeoffs that I typically don't have access to: accuracy-speed[9] or accuracy-space. There have been too many times that I've had to say, "I wish I could do this, but it would take too much time/space to compute." Most of these problems still work even if the accuracy is not 100%. And furthermore, many (if not all of these) can tune accuracy to by parameter adjustment anyways. They tend to have favorable combinatorial properties ie: they form monoids or semigroups under merge operations. In short, a property of data structures that gave me the ability to solve problems I couldn't before.

I hope they are as useful or intriguing to you as they are to me.

1. https://github.com/tdunning/t-digest

2. https://pdsa.readthedocs.io/en/latest/rank/qdigest.html

3. https://florian.github.io/count-min-sketch/

4. https://www.cs.yale.edu/homes/el327/papers/simpleMatrixSketc...

5. https://www.juanlopes.net/poly18/poly18-juan-lopes.pdf

6. https://courses.engr.illinois.edu/cs498abd/fa2020/slides/20-...

7. https://people.csail.mit.edu/rrw/6.045-2017/encalgs-mg.pdf

8. https://www.sciencedirect.com/science/article/abs/pii/S00200...

9. It may better be described as error-speed and error-space, but I've avoided the term error because the term for programming audiences typically evokes the idea of logic errors and what I mean is statistical error.

Ah, I misunderstood what you meant. If you are reporting static buckets I get how that is better than what folks typically do but how do you know the buckets a priori? Others back their histograms with things like https://github.com/tdunning/t-digest. It is pretty powerful as the buckets are dynamic based on the data and histograms can be added together.

Yes, but if you need to estimate median on the fly (e.g., over a stream of data) or in parallel there are better ways.

There are some newer data structures that take this to the next level such as T-Digest[1], which remains extremely accurate even when determining percentiles at the very tail end (like 99.999%)

[1]: https://arxiv.org/pdf/1902.04023.pdf / https://github.com/tdunning/t-digest

[2] https://github.com/tdunning/t-digest

T-Digest

This is pretty cool. The title would be a bit more descriptive if it were “Fast Rolling Quantile Filters for Python”, since the high-pass/low-pass filter functionality seems to be the focus.

The README mentions it uses binary heaps - if you’re willing to accept some (bounded) approximation, then it should be possible to reduce memory usage and somewhat reduce runtime by using a sketching data structure like Dunning’s t-digest: https://github.com/tdunning/t-digest/blob/main/docs/t-digest....

There is an open source Python implementation, although I haven’t used it and can’t vouch for its quality: https://github.com/CamDavidsonPilon/tdigest