LakeSoul
starrocks
LakeSoul | starrocks | |
---|---|---|
21 | 12 | |
2,307 | 7,789 | |
0.8% | 2.2% | |
9.2 | 10.0 | |
7 days ago | about 22 hours ago | |
Java | Java | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
LakeSoul
-
Open Source first Anniversary Star 1.2K! Review on the anniversary of LakeSoul, the unique open-source Lakehouse
Review code reference: https://github.com/meta-soul/LakeSoul/pull/115
-
The best Open-source lakehouse project, LakeSoul 2.0, supports snapshot, rollback, Flink, and Hive interconnection
In LakeSoul 2.0, metadata and database interaction are fully implemented using the Postgres SQL (PG) protocol for reasons mentioned at https://github.com/meta-soul/LakeSoul/issues/23. On the one hand, Cassandra does not support single-table multi-partition transactions. On the other hand, Cassandra cluster management has higher maintenance costs, while the Postgres SQL protocol is widely used in enterprises and has lower maintenance costs. You need to configure PG parameters. For details, click https://github.com/meta-soul/LakeSoul/wiki/02.-QuickStart
-
A New One-stop AI development and production platform, AlphaIDE
I’ve posted about LakeSoul, an open-source framework for unified streaming and batch table storage, and MetaSpore, an open-source platform for machine learning.
-
Build a real-time machine learning sample library using the best open-source project about big data and data lakehouse, LakeSoul
2.4 Data Backfill Since LakeSoul supports Upsert of any Range partitioned data, there is no difference between backtracking and streaming write. When the data to be inserted is ready, Spark performs Upsert to update historical data. LakeSoul automatically recognizes Schema changes. Update meta information of tables to implement Schema evolution. LakeSoul provides a complete storage function of data warehouse tables, and each historical partition can be queried and updated. Compared with Flink’s window Join scheme, it solves the problem of invisible intermediate states and can quickly realize mass updates and traceability of historical data.
The previous article, "The design concept of the best open-source project about big data and data lakehouse" introduced the design concept and partial realization principle of LakeSoul's open-source and stream batch integrated surface storage framework. The original intention of the design of LakeSoul is to solve various problems that are difficult to solve in traditional Hive data warehouse scenarios, including Upsert update, Merge on Read, and concurrent write. This article will demonstrate the core capabilities of LakeSoul using a typical application scenario: building a real-time machine learning sample library.
-
Solved a practical business problem when using Hudi: LakeSoul supports null field non-override semanticssemantics
Recently, the LakeSoul r&d team helped users solve a practical business problem using Hudi. Here is a summary and record. The business process is that the upstream system extracts the original data from the online DB table into JSON format and writes it into Kafka. The downstream system uses Spark to read the messages in Kafka. The data is updated and aggregated using Hudi and sent to the downstream database for analysis.
-
What is the Lakehouse, the latest Direction of Big Data Architecture?
Lakesoul
-
Design concept of a best opensource project about big data and data lakehouse
LakeSoul is a streaming batch integrated table storage framework developed by DMetaSoul, which has made a lot of design optimization around the new trend of big data architecture systems. This paper explains the core concept and design principle of LakeSoul, the Open-source Project, in detail.
-
Data engine engineers interview for help
Maybe you can use some of this code with a dataset over the next two days and compare the products to show the interviewer that you know a lot about the projects. Interviewers like candidates who can easily tell the difference between different products. Perhaps take a look at Lakesoul, similar to Iceberg, Hudi, etc., whose GitHub has a comparison of open-source data lake projectsand how to use them. You can also check out Iceberg, Hudi's website, which has detailed tutorials.
-
Details of 4 best opensource projects about big data you should try out(Ⅰ)
1.Introduction LakeSoul is a streaming batch integrated table storage framework built on The Apache Spark engine. It has highly extensible metadata management, ACID transactions, efficient and flexible UPSERT operations, Schema evolution, and batch integration processing. LakeSoul specifically optimizes the row and column level incremental updates, high concurrent entries, and batch scan reads for data on top of the Data Lake cloud storage. The storage separation architecture of cloud-native computing makes deployment very simple while supporting huge data volumes at a very low cost. LakeSoul supports high-performance write throughput in hashed partition primary key UPsert scenarios through lSM-tree, which can reach 30MB/s/core on object storage systems such as S3. The highly optimized Merge on Reading implementation also ensures Read performance. LakeSoul manages metadata through Cassandra to achieve high scalability of metadata. LakeSoul’s main features are as follows:
starrocks
-
A MySQL compatible database engine written in pure Go
tidb has been around for a while, it is distributed, written in Go and Rust, and MySQL compatible. https://github.com/pingcap/tidb
Somewhat relatedly, StarRocks is also MySQL compatible, written in Java and C++, but it's tackling OLAP use-cases. https://github.com/StarRocks/starrocks
- StarRocks – sub-second MPP OLAP database for full analytics scenarios
-
Let's Talk about Joins
I think you're talking about doing denormalization before importing data into an OLAP system to avoid subsequent joins. However, this greatly limits the flexibility of data modeling. Moreover, denormalization can be a headache-inducing process. In fact, I have tested StarRocks (https://github.com/StarRocks/starrocks), and it is capable of performing joins while streaming data imports, and the speed is very fast. It's worth giving it a try.
-
Ask HN: Are there any notable Chinese FLOSS projects?
https://github.com/apache/doris Is a great example. Same for it's cousin https://github.com/StarRocks/starrocks that was an early fork of the doris project.
To be fair, these are the only examples I can think of and I only learned of these as I'm standing up new data infra using starrocks.
-
Open Source Columnar Databases
ClickHouseClickHouse and Starrocks are similar. They are both columnar databases powered by vectorization tech, which means they are really fast.
-
Ask HN: Do you use any software (mainly) developed in China?
StarRocks, it’s a Linux Foundation project now, but a lot of the initial team and community behind it came from China.
https://github.com/StarRocks/starrocks
Funny that I hadn’t heard of them in the database space till they showed up at the top of ClickBench. Makes me wonder what other interesting projects I’m missing out on in China.
- Anyone using StarRocks DB instead of ClickHouse?
-
Show HN: A benchmark for analytical databases (Snowflake, Druid, Redshift)
Full disclosure - I work for StarRocks (starrocks.com)
First of all, this is great. Transparent and healthy competition is always great for the customers!
Regarding the joined table queries that are missing in the tests, this is exactly why we built StarRocks - to give people the best performance of complex analytics queries on both joined tables and single tables.
I encourage you to checkout this blog: https://starrocks.medium.com/starrocks-outperforms-clickhous...
And, give us a star if you think we are doing the right thing: https://github.com/StarRocks/starrocks
Follow us on LinkedIn for the latest updates: https://www.linkedin.com/company/starrocks
- We are looking for a very fast database for big data analysis, does anyone know about starrocks, I heard it is very fast
- wow, i found a super fast database for Big Data analytics,it's called StarRocks,come and take a look!
What are some alternatives?
MetaSpore - A unified end-to-end machine intelligence platform
ClickBench - ClickBench: a Benchmark For Analytical Databases
iceberg - Apache Iceberg
doris - Apache Doris is an easy-to-use, high performance and unified analytics database.
delta - An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
duckdb - DuckDB is an in-process SQL OLAP Database Management System
hudi - Upserts, Deletes And Incremental Processing on Big Data.
TablePlus - TablePlus macOS issue tracker
delta-sharing - An open protocol for secure data sharing
clickhouse-bulk - Collects many small inserts to ClickHouse and send in big inserts
nussknacker - Low-code tool for automating actions on real time data | Stream processing for the users.
ClickHouse - ClickHouse® is a free analytics DBMS for big data