duckdb VS Rx.NET

I'm not sure if the fix is reassuring or not: https://github.com/duckdb/duckdb/pull/9411/files

It's an interesting question!

The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.

It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.

[1] https://github.com/bacalhau-project/bacalhau

[2] https://github.com/duckdb/duckdb

[4] Switch to Push-Based Execution Model by Mytherin · Pull Request #2393 · duckdb/duckdb (github.com)

it depends on your query obviously.

In general, I did very deep benchmarking of pg, clickhouse and duckdb, and I sure didn't make stupid mistakes like this: https://news.ycombinator.com/item?id=36990831

My dataset has 50B rows and 2tb of data, and I think columnar dbs are very overhiped and I chose pg because:

- pg performance is acceptable, maybe 2-3x times slower than clickhouse and duckdb on some queries if pg is configured correctly and run on compressed storage

- clickhouse and duckdb start falling apart very fast because they specialized on very narrow type of queries: https://github.com/ClickHouse/ClickHouse/issues/47520 https://github.com/ClickHouse/ClickHouse/issues/47521 https://github.com/duckdb/duckdb/discussions/6696

This action installs duckdb with the version provided in input.

Duckdb: https://github.com/duckdb/duckdb - seems pretty popular, been keeping an eye on this for close to a year now.

The Parquet-Go library is very complex, not yet success to use it. So I ask whether DuckDB can provide API https://github.com/duckdb/duckdb/issues/7776

One of the greatest things i discovered recently, is Reactive programming / Reactive Extensions ( https://github.com/dotnet/reactive ).

https://github.com/dotnet/reactive has a lot of different time related extensions for "events". Maybe you'll find something for yourself, if you google for rate limiting with reactive.

Install Reactive Extensions: https://github.com/dotnet/reactive

Another option is to use the RxJava library in Java. This library uses reactive programming principles to make it easy to write asynchronous and event-driven code. It's particularly well-suited for handling streams of data and allows you to write code that is both efficient and easy to read.

I'd use a dedicated event bus based on Reactive Extensions or MediatR to publish domain events from your domain services. This probably doesn't solve all your ViewModel update problems as is, maybe you need to revise the granularity (maybe you can have smaller ViewModels that refresh single property that exposes the Model) and lifespan (sometimes you can create a ViewModel, make it perform it's task and then discard it completely) of your ViewModels.

If you're finding yourself in a situation where you need to turn this behavior into a pattern because there are a lot of View Models that need to execute async business logic in response to some changes, I'd go with something like MediatR or Reactive Extensions. The idea is, again, that some other, probably business-level, component listens to changes in a decoupled way (that means it doesn't subscribe directly to your View Model, but to an event bus instead). View Model publishes change events to the event bus, and business-component reacts to these events by executing the business logic.

We'd really appreciate if it consumers of the library could update and provide any issues / bugs via the GitHub repo: https://github.com/dotnet/reactive/issues