shardingsphere-elasticjob-ui VS doris

ShardingSphere claims to offer an ecosystem able to transform any database into a distributed database system. It acts as a proxy between your code and your database(s). It comes in two flavors:

The application uses Apache Shardingsphre to route again depending on the data. If the value computed by the API Gateway is correct, the flow stays "in its lane"; if not, it's routed to the correct database, but with a performance penalty as it's outside its lane.

Apache ShardingSphere

Apache ShardingSphere supports data encryption. By parsing users’ SQL input and rewriting the SQL according to the users’ encryption rules, the original data is encrypted and stored with ciphertext data in the underlying database at the same time.

Apache ShardingSphere is a distributed database ecosystem that can transform any database into a distributed database system, and enhance it with sharding, elastic scaling, encryption features & more.

ShardingSphere supports a database gateway, but its heterogeneous capability is limited to the logical database in previous versions. This means that all the data sources under a logical database must be of the same database type.

ShardingSphere Official Website

ElasticJob UI is the visual admin console of ElasticJob, whose target users are developers and DevOps teams rather than users. Generally, it is deployed only in the internal environment and thus its R&D focus more on its features.

As an open-source real-time data warehouse, Apache Doris provides semi-structured data processing capabilities, and the newly-released version 2.1.0 makes a stride in this direction. Before V2.1, Apache Doris stores semi-structured data as JSON files. However, during query execution, the real-time parsing of JSON data leads to high CPU and I/O consumption in addition to high query latency, especially when the dataset is huge and complicated. Moreover, the lack of a pre-defined schema means there is no handle for query optimization.

Apache Doris is a real-time data warehouse.

Learn more about Apache Doris or find the Doris makers on Slack.

As you can imagine, a long and complicated data pipeline is high-maintenance and detrimental to development efficiency. Moreover, they are not capable of ad-hoc queries. So as an upgrade to our data warehouse, we replaced most of these components with Apache Doris, a unified analytic database.

GitHub source code: https://github.com/apache/doris/tree/branch-2.0

The key to Architecture 3.0 is the combination of Flink and Doris, so this is how to connect them. Probably the most important code in building architecture 3. flink-demo stream-load-demo

https://github.com/apache/doris Is a great example. Same for it's cousin https://github.com/StarRocks/starrocks that was an early fork of the doris project.

To be fair, these are the only examples I can think of and I only learned of these as I'm standing up new data infra using starrocks.

We optimized Apache Doris to solve these problems. (Pull Request on Github)

So, in Storage Architecture 2.0, we introduced Apache Doris and Apache Spark. The whole data pipeline was a Y-shaped diagram.