doris VS sanitizers

As an open-source real-time data warehouse, Apache Doris provides semi-structured data processing capabilities, and the newly-released version 2.1.0 makes a stride in this direction. Before V2.1, Apache Doris stores semi-structured data as JSON files. However, during query execution, the real-time parsing of JSON data leads to high CPU and I/O consumption in addition to high query latency, especially when the dataset is huge and complicated. Moreover, the lack of a pre-defined schema means there is no handle for query optimization.

Apache Doris is a real-time data warehouse.

Learn more about Apache Doris or find the Doris makers on Slack.

As you can imagine, a long and complicated data pipeline is high-maintenance and detrimental to development efficiency. Moreover, they are not capable of ad-hoc queries. So as an upgrade to our data warehouse, we replaced most of these components with Apache Doris, a unified analytic database.

GitHub source code: https://github.com/apache/doris/tree/branch-2.0

The key to Architecture 3.0 is the combination of Flink and Doris, so this is how to connect them. Probably the most important code in building architecture 3. flink-demo stream-load-demo

https://github.com/apache/doris Is a great example. Same for it's cousin https://github.com/StarRocks/starrocks that was an early fork of the doris project.

To be fair, these are the only examples I can think of and I only learned of these as I'm standing up new data infra using starrocks.

We optimized Apache Doris to solve these problems. (Pull Request on Github)

So, in Storage Architecture 2.0, we introduced Apache Doris and Apache Spark. The whole data pipeline was a Y-shaped diagram.

AddressSanitizer is really useful, it's similar to Valgrind but has much lower overhead.

And if you're up for it, I'd further recommend adding some ways to deal with buffer overflows in debug builds. The way I deal with this is by using Address-Sanitizer's manual poisoning api. Bonus point if you leave additional poisoned space between allocations so off by one errors are likely to end up in a poisoned region instead of nearby allocation.

It also doesn't hurt that Miri can find many kinds of unsafe violations even in unsafe blocks. Zig may get something like this one day, but even if it does, checking things at runtime is not a substitute for compile time -- the C++ Sanitizers haven't exactly solved the safety story for C++ even over a decade later.

This is where stuff like ASan is really useful.

Use: https://github.com/google/sanitizers/wiki/AddressSanitizer

Have you tried using tools like ASAN/LSAN or valgrind to confirm that there are indeed no memory leaks?

Compile with ASAN and UBSAn