SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Java Big Data Projects
-
7. Apache Flink | Github | tutorial
-
We have some of this functionality in Presto (https://github.com/prestodb/presto), but it takes fair bit of work to implement it for all the different backends.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
If your data lacks uniform time intervals between consecutive entries, QuestDB offers a solution by allowing you to sample your data. After that, MindsDB facilitates creating, training, and deploying your time-series models.
-
Trino
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
Project mention: Trino: Fast distributed SQL query engine for big data analytics | news.ycombinator.com | 2024-03-19 -
-
starrocks
StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. InfoWorld’s 2023 BOSSIE Award for best open source software.
Project mention: A MySQL compatible database engine written in pure Go | news.ycombinator.com | 2024-04-09tidb has been around for a while, it is distributed, written in Go and Rust, and MySQL compatible. https://github.com/pingcap/tidb
Somewhat relatedly, StarRocks is also MySQL compatible, written in Java and C++, but it's tackling OLAP use-cases. https://github.com/StarRocks/starrocks
-
Project mention: Ask HN: Does (or why does) anyone use MapReduce anymore? | news.ycombinator.com | 2024-01-24
The "streaming systems" book answers your question and more: https://www.oreilly.com/library/view/streaming-systems/97814.... It gives you a history of how batch processing started with MapReduce, and how attempts at scaling by moving towards streaming systems gave us all the subsequent frameworks (Spark, Beam, etc.).
As for the framework called MapReduce, it isn't used much, but its descendant https://beam.apache.org very much is. Nowadays people often use "map reduce" as a shorthand for whatever batch processing system they're building on top of.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Zeppelin
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Now we can proceed with the definition of Apache Zeppelin. It is a web-based notebook that enables data-driven, interactive data analytics and collaborative documents with Python, Scala, SQL, Spark, and more. You can execute code and even schedule a job (via cron) to run at regular intervals.
-
Hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
Project mention: Does anyone know any good java implementations for distributed key-value store? | /r/ExperiencedDevs | 2023-06-08You're probably looking for Hazelcast here. Note that it does much more than just a distributed k/v, but it will get you where you need to go.
-
Vespa(4.3k ⭐) → A fully featured search engine and vector database. It supports vector search (ANN), lexical search, and search in structured data, all in the same query. Integrated machine-learned model inference allows you to apply AI to make sense of your data in real time.
-
-
-
Project mention: Data diffs: Algorithms for explaining what changed in a dataset (2022) | news.ycombinator.com | 2023-07-26
> Make diff work on more than just SQLite.
Another way of doing this that I've been wanting to do for a while is to implement the DIFF operator in Apache Calcite[0]. Using Calcite, DIFF could be implemented as rewrite rules to generate the appropriate SQL to be directly executed against the database or the DIFF operator can be implemented outside of the database (which the original paper shows is more efficient).
-
-
Crate
CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.
-
-
-
-
LakeSoul
LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.
-
Project mention: Git Query Language (GQL) Aggregation Functions, Groups, Alias | /r/ProgrammingLanguages | 2023-06-30
Also are you familiar with apache drill . The idea is to put an SQL interpreter in front of any kind of database just like you are doing for git here.
-
bookkeeper
Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads
-
paimon
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
18. Apache Paimon | Github | tutorial
-
Project mention: Summing columns in remote Parquet files using DuckDB | news.ycombinator.com | 2023-11-16
Right, there's all sorts of metadata and often stats included in any parquet file: https://github.com/apache/parquet-format#file-format
The offsets of said metadata are well-defined (i.e. in the footer) so for S3 / blob storage so long as you can efficiently request a range of bytes you can pull the metadata without having to read all the data.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2024-04-09.
Java Big Data related posts
- Top 10 Common Data Engineers and Scientists Pain Points in 2024
- Linkedin OpenHouse: Control Plane for Tables in Data Lakehouses
- Choosing Between a Streaming Database and a Stream Processing Framework in Python
- Ask HN: Does (or why does) anyone use MapReduce anymore?
- StarRocks – sub-second MPP OLAP database for full analytics scenarios
- Five Apache projects you probably didn't know about
- Getting Started with Flink SQL, Apache Iceberg and DynamoDB Catalog
-
A note from our sponsor - SaaSHub
www.saashub.com | 18 Apr 2024
Index
What are some of the best open-source Big Data projects in Java? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Apache Flink | 23,128 |
| 2 | Presto | 15,582 |
| 3 | QuestDB | 13,420 |
| 4 | Trino | 9,519 |
| 5 | kafka-ui | 8,416 |
| 6 | starrocks | 7,726 |
| 7 | beam | 7,477 |
| 8 | Zeppelin | 6,261 |
| 9 | Hazelcast | 5,853 |
| 10 | vespa | 5,323 |
| 11 | Apache Hive | 5,320 |
| 12 | Apache Ignite | 4,675 |
| 13 | Apache Calcite | 4,344 |
| 14 | iotdb | 4,249 |
| 15 | Crate | 3,952 |
| 16 | fastjson2 | 3,411 |
| 17 | Flume | 2,503 |
| 18 | Apache Parquet | 2,398 |
| 19 | LakeSoul | 2,294 |
| 20 | Apache Drill | 1,891 |
| 21 | bookkeeper | 1,845 |
| 22 | paimon | 1,792 |
| 23 | parquet-format | 1,633 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com