nebula VS uPlot

we use our in-house baked engine - open sourced here https://github.com/varchar-io/nebula

Yeah, Tinybird has lots of similarities, I will do more research on it, thanks for the reference.

Take a look at this open source project - it may be helpful - https://github.com/varchar-io/nebula

cool - I also hand crafted a CSV parser following RFC4180 a while ago, not sure if you have a repeatable way to benchmark the performance difference?

https://github.com/varchar-io/nebula/blob/master/src/storage...

- if you look for hosting solution by yourself, bringing up a nebula cluster (even a single node) is simple, check out https://github.com/varchar-io/nebula

Many modern data systems are designed in a similar way - extract, index, and query in including many low-latency real-time analytical systems, such as clickhouse, druid, pinot, nebula. Take nebula (https://github.com/varchar-io/nebula) as an example - it connects real-time storage engines like Kafka, cloud storage, or pubsub systems, extracts and index data from message queue into its own distributed system, providing low latency query on top of it for business use cases.

Here is a very efficient 2D canvas lib that is extremely efficient https://github.com/leeoniya/uPlot i recommend looking through their readme and seeing if it solves your current problems.

BACKEND:

It's about 3000 lines of Crystal code https://crystal-lang.org/ -- it's been an absolute dream to program in. Ruby-like syntax, statically compiled performance. Nice standard library. Would recommend :)

Performance == caching

Local ephemeral filesystem for shortest-term caching.

Redis for shared medium-term caching and locking.

S3-compatible object storage for longest-term caching (for raw data feeds pulled every evening by a cronjob).

FRONTEND:

Bootstrap

uPlot https://github.com/leeoniya/uPlot

Just a sprinkle of inline JS+CSS on the page.

you can use matchMedia to detect devicePixelRatio changes and re-sync:

https://github.com/leeoniya/uPlot/blob/190134aa844cfa2a0c052...

everything stays crisp even as you browser-zoom. e.g. https://leeoniya.github.io/uPlot/demos/area-fill.html

Safari's slow release cadence (typically tied to OS version upgrades) does leave it more broken and for longer than Firefox, Chrome, Opera and other browsers that are updated more frequently and not tied to the OS.

just recently i ran into incompatibilities with Safari < 14 needing the old-spec matchmedia api:

https://github.com/leeoniya/uPlot/issues/538#issuecomment-87...

uPlot will handle this. https://github.com/leeoniya/uPlot

If you are showing 100s of thousands of points, I'd still go with canvas eg uPlot. It's a really, really excellent library.

The existing node red chart tools is not great. I'd like to be able to use uplot instead. The uplot-charts node should work very much like that charts node built into node-red-dashboard.

super cool!

i've done my share of optimizations and benchmarking charting libs [1] but this goes deeper still.

SVG is not a good choice for large raw datasets when you need performance.

[1] https://github.com/leeoniya/uPlot#performance