doris
flink-kubernetes-operator
doris | flink-kubernetes-operator | |
---|---|---|
42 | 7 | |
11,363 | 718 | |
1.6% | 3.3% | |
10.0 | 9.2 | |
5 days ago | 4 days ago | |
Java | Java | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
doris
-
Variant in Apache Doris 2.1.0: a new data type 8 times faster than JSON for semi-structured data analysis
As an open-source real-time data warehouse, Apache Doris provides semi-structured data processing capabilities, and the newly-released version 2.1.0 makes a stride in this direction. Before V2.1, Apache Doris stores semi-structured data as JSON files. However, during query execution, the real-time parsing of JSON data leads to high CPU and I/O consumption in addition to high query latency, especially when the dataset is huge and complicated. Moreover, the lack of a pre-defined schema means there is no handle for query optimization.
-
Five Apache projects you probably didn't know about
Apache Doris is a real-time data warehouse.
-
Log Analysis: Elasticsearch VS Apache Doris
Learn more about Apache Doris or find the Doris makers on Slack.
-
Replacing Apache Hive, Elasticsearch, and PostgreSQL With Apache Doris
As you can imagine, a long and complicated data pipeline is high-maintenance and detrimental to development efficiency. Moreover, they are not capable of ad-hoc queries. So as an upgrade to our data warehouse, we replaced most of these components with Apache Doris, a unified analytic database.
-
Apache Doris 2.0 Beta Now Available: Faster, Stabler, and More Versatile
GitHub source code: https://github.com/apache/doris/tree/branch-2.0
-
A/B Testing was a handful
The key to Architecture 3.0 is the combination of Flink and Doris, so this is how to connect them. Probably the most important code in building architecture 3. flink-demo stream-load-demo
-
Ask HN: Are there any notable Chinese FLOSS projects?
https://github.com/apache/doris Is a great example. Same for it's cousin https://github.com/StarRocks/starrocks that was an early fork of the doris project.
To be fair, these are the only examples I can think of and I only learned of these as I'm standing up new data infra using starrocks.
- Apache Doris 2.0.0 Alpha Released
-
30,000 QPS Per Node: How We Increased Database Query Concurrency by 20 Times
We optimized Apache Doris to solve these problems. (Pull Request on Github)
-
Beginner's Guide to Data Analytics: Diving into Our Data Management Platform
So, in Storage Architecture 2.0, we introduced Apache Doris and Apache Spark. The whole data pipeline was a Y-shaped diagram.
flink-kubernetes-operator
-
Top 10 Common Data Engineers and Scientists Pain Points in 2024
Data scientists often prefer Python for its simplicity and powerful libraries like Pandas or SciPy. However, many real-time data processing tools are Java-based. Take the example of Kafka, Flink, or Spark streaming. While these tools have their Python API/wrapper libraries, they introduce increased latency, and data scientists need to manage dependencies for both Python and JVM environments. For example, implementing a real-time anomaly detection model in Kafka Streams would require translating Python code into Java, slowing down pipeline performance, and requiring a complex initial setup.
-
Choosing Between a Streaming Database and a Stream Processing Framework in Python
Other stream processing engines (such as Flink and Spark Streaming) provide SQL interfaces too, but the key difference is a streaming database has its storage. Stream processing engines require a dedicated database to store input and output data. On the other hand, streaming databases utilize cloud-native storage to maintain materialized views and states, allowing data replication and independent storage scaling.
- FLaNK Stack Weekly 22 January 2024
-
Go concurrency simplified. Part 4: Post office as a data pipeline
also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc.
-
Five Apache projects you probably didn't know about
Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around Apache Spark or Apache Flink. Be careful, as each engine comes with its own set of features.
-
Getting Started with Flink SQL, Apache Iceberg and DynamoDB Catalog
Due to the technology transformation we want to do recently, we started to investigate Apache Iceberg. In addition, the data processing engine we use in house is Apache Flink, so it's only fair to look for an experimental environment that integrates Flink and Iceberg.
- FLaNK Stack Weekly for 07August2023
What are some alternatives?
starrocks - StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. InfoWorld’s 2023 BOSSIE Award for best open source software.
hugging-chat-api - HuggingChat Python API🤗
tools
anomalib - An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
Trino - Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
ToolBench - [ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.
kop - Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar
CallCMLModel - An example on calling models deployed in CML
Boost-Pretty-Printer - GDB Pretty Printers for Boost
Qwen-7B - The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud. [Moved to: https://github.com/QwenLM/Qwen]
esphome-yeelight-ceiling-light - ESPHome custom firmware for some Yeelight Ceiling Lights
cdf-workshop