nussknacker
LakeSoul
Our great sponsors
nussknacker | LakeSoul | |
---|---|---|
1 | 21 | |
609 | 2,307 | |
5.6% | 2.0% | |
9.8 | 9.2 | |
5 days ago | 4 days ago | |
Scala | Java | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
nussknacker
LakeSoul
-
Open Source first Anniversary Star 1.2K! Review on the anniversary of LakeSoul, the unique open-source Lakehouse
Review code reference: https://github.com/meta-soul/LakeSoul/pull/115
-
The best Open-source lakehouse project, LakeSoul 2.0, supports snapshot, rollback, Flink, and Hive interconnection
In LakeSoul 2.0, metadata and database interaction are fully implemented using the Postgres SQL (PG) protocol for reasons mentioned at https://github.com/meta-soul/LakeSoul/issues/23. On the one hand, Cassandra does not support single-table multi-partition transactions. On the other hand, Cassandra cluster management has higher maintenance costs, while the Postgres SQL protocol is widely used in enterprises and has lower maintenance costs. You need to configure PG parameters. For details, click https://github.com/meta-soul/LakeSoul/wiki/02.-QuickStart
-
A New One-stop AI development and production platform, AlphaIDE
I’ve posted about LakeSoul, an open-source framework for unified streaming and batch table storage, and MetaSpore, an open-source platform for machine learning.
-
Build a real-time machine learning sample library using the best open-source project about big data and data lakehouse, LakeSoul
2.4 Data Backfill Since LakeSoul supports Upsert of any Range partitioned data, there is no difference between backtracking and streaming write. When the data to be inserted is ready, Spark performs Upsert to update historical data. LakeSoul automatically recognizes Schema changes. Update meta information of tables to implement Schema evolution. LakeSoul provides a complete storage function of data warehouse tables, and each historical partition can be queried and updated. Compared with Flink’s window Join scheme, it solves the problem of invisible intermediate states and can quickly realize mass updates and traceability of historical data.
The previous article, "The design concept of the best open-source project about big data and data lakehouse" introduced the design concept and partial realization principle of LakeSoul's open-source and stream batch integrated surface storage framework. The original intention of the design of LakeSoul is to solve various problems that are difficult to solve in traditional Hive data warehouse scenarios, including Upsert update, Merge on Read, and concurrent write. This article will demonstrate the core capabilities of LakeSoul using a typical application scenario: building a real-time machine learning sample library.
-
Solved a practical business problem when using Hudi: LakeSoul supports null field non-override semanticssemantics
Recently, the LakeSoul r&d team helped users solve a practical business problem using Hudi. Here is a summary and record. The business process is that the upstream system extracts the original data from the online DB table into JSON format and writes it into Kafka. The downstream system uses Spark to read the messages in Kafka. The data is updated and aggregated using Hudi and sent to the downstream database for analysis.
-
What is the Lakehouse, the latest Direction of Big Data Architecture?
Lakesoul
-
Design concept of a best opensource project about big data and data lakehouse
LakeSoul is a streaming batch integrated table storage framework developed by DMetaSoul, which has made a lot of design optimization around the new trend of big data architecture systems. This paper explains the core concept and design principle of LakeSoul, the Open-source Project, in detail.
-
Data engine engineers interview for help
Maybe you can use some of this code with a dataset over the next two days and compare the products to show the interviewer that you know a lot about the projects. Interviewers like candidates who can easily tell the difference between different products. Perhaps take a look at Lakesoul, similar to Iceberg, Hudi, etc., whose GitHub has a comparison of open-source data lake projectsand how to use them. You can also check out Iceberg, Hudi's website, which has detailed tutorials.
-
Details of 4 best opensource projects about big data you should try out(Ⅰ)
1.Introduction LakeSoul is a streaming batch integrated table storage framework built on The Apache Spark engine. It has highly extensible metadata management, ACID transactions, efficient and flexible UPSERT operations, Schema evolution, and batch integration processing. LakeSoul specifically optimizes the row and column level incremental updates, high concurrent entries, and batch scan reads for data on top of the Data Lake cloud storage. The storage separation architecture of cloud-native computing makes deployment very simple while supporting huge data volumes at a very low cost. LakeSoul supports high-performance write throughput in hashed partition primary key UPsert scenarios through lSM-tree, which can reach 30MB/s/core on object storage systems such as S3. The highly optimized Merge on Reading implementation also ensures Read performance. LakeSoul manages metadata through Cassandra to achieve high scalability of metadata. LakeSoul’s main features are as follows:
What are some alternatives?
flink-faker - A data generator source connector for Flink SQL based on data-faker.
MetaSpore - A unified end-to-end machine intelligence platform
flink-on-k8s-operator - Kubernetes operator for managing the lifecycle of Apache Flink and Beam applications.
iceberg - Apache Iceberg
yauaa - Yet Another UserAgent Analyzer
delta - An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Scio - A Scala API for Apache Beam and Google Cloud Dataflow.
hudi - Upserts, Deletes And Incremental Processing on Big Data.
kafka-manager - CMAK is a tool for managing Apache Kafka clusters
delta-sharing - An open protocol for secure data sharing
Gearpump - Lightweight real-time big data streaming engine over Akka
starrocks - StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. InfoWorld’s 2023 BOSSIE Award for best open source software.