Top 23 Java Big Data Projects
-
Project mention: Avro SpecificRecord File Sink using apache flink is not compiling due to error incompatible types: FileSink<?> cannot be converted to SinkFunction<?> | reddit.com/r/apacheflink | 2021-09-14
[1]: https://mvnrepository.com/artifact/org.apache.avro/avro-maven-plugin/1.8.2 [2]: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java [3]: https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/file_sink/ [4]: https://github.com/apache/flink/blob/c81b831d5fe08d328251d91f4f255b1508a9feb4/flink-end-to-end-tests/flink-file-sink-test/src/main/java/FileSinkProgram.java [5]: https://github.com/rajcspsg/streaming-file-sink-demo
-
Project mention: Let's write a compiler, part 5: A code generator | news.ycombinator.com | 2021-08-19
-
-
-
Zeppelin
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
Project mention: What libraries do you use for machine learning and data visualizing in scala? | reddit.com/r/scala | 2021-11-27Another more widely used notebooks for scala and spark: https://zeppelin.apache.org/
-
Apache Beam
-
Project mention: Show HN: Hazelcast 5 BETA – streaming+storage in one | news.ycombinator.com | 2021-07-16
-
Trino
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
You might find https://trino.io/ interesting. It allows you to bolt on a MPP SQL execution engine on top of any data source including pre-built connectors for Druid and Kafka.
It's all ANSI SQL and the best part is you can combine data from heterogenous sources. e.g. You can join data between a topic in Kafka and a table in Druid or even between Kafka, S3 and your RDBMS.
Disclaimer: I'm a maintainer of the project.
-
Scout APM
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
-
An important requirement for data privacy and protection is to find and catalog tables and columns that contain PII or PHI data in a data warehouse. Open source data catalogs like Datahub and Amundsen enable cataloging of information in data warehouses. Moreover, tables and columns can be tagged including PII and type of PII tags.
-
Project mention: .NET and Apache Ignite: Testing Cache and SQL API features — Part I | reddit.com/r/luisdev | 2021-09-11
Last days, I started using Apache Ignite as a cache strategy for some applications. Apache Ignite is an open-source In-Memory Data Grid, distributed database, caching, and high-performance computing platform.
-
Apache Hive takes in a specific SQL dialect and converts it to map-reduce.
-
Project mention: MeiliSearch: A Minimalist Full-Text Search Engine | news.ycombinator.com | 2021-08-15
After looking at various alternatives, I'm thinking of trying out https://vespa.ai/ [0]
-
Project mention: Anyone know of any software that can help in designing then outputting to various database | reddit.com/r/DatabaseHelp | 2021-11-21
Abstraction Layer - You can use something like Calcite to abstract out your data storage. https://calcite.apache.org/
-
Flume
-
Project mention: Apache Drill: the reports of my death have been greatly exaggerated | news.ycombinator.com | 2021-11-01
>We’ve started talking about speeding up our release cadence to better reflect our recent activity.
There's been only one release per year in the past so you can't fault anyone to think the project is dead.
-
Project mention: Scalable, fault-tolerant, low-latency storage service for real-time workloads | news.ycombinator.com | 2021-10-26
-
Hi, I've been trying to write parquet files on android for the past couple of days, and have really been struggling to find a solution. My original hypothesis was to just use the java parquet implementation (https://github.com/apache/parquet-mr), but I've since realized that not all java libraries play well with Android. I've gone through essentially dependency hell trying to franken-fit the library into my project, and imported as much as i could before hitting walls such as this one (https://github.com/mockito/mockito/issues/841).
-
DatumBox
Datumbox is an open-source Machine Learning framework written in Java which allows the rapid development of Machine Learning and Statistical applications.
-
Project mention: Build your own “data lake” for reporting purposes | news.ycombinator.com | 2021-03-14
For my home projects I generate parquet (columnar and very well suited for DW like queries) files with pyarrow and use https://github.com/dremio/dremio-oss (https://www.dremio.com/on-prem/) to query them on lake (minio or just local disk or s3) and use Apache Superset for quick charts or dashboards.
-
Hazelcast Jet
-
-
-
Rakam
📈 Collect customer event data from your apps. (Note that this project only includes the API collector, not the visualization platform)
Project mention: Show HN: Lightdash – An open source Looker alternative | news.ycombinator.com | 2021-06-03Well done! I was actually looking for an open source LookML a while back and found Rakam[0]. It seems they added the dbt layer after the fact while you started with that concept. Product looks slick, good luck?
By the way, what happened with Hubble?
-
Project mention: Proiectele open source la care am lucrat în ultimă perioada | reddit.com/r/programare | 2021-08-01
www.mockneat.com - o librărie Java care permite crearea de date aleatoare, utila in testarea de aplicații.
Java Big Data related posts
- The Next Big Challenge for Data Is Organizational
- Apache Drill: the reports of my death have been greatly exaggerated
- Launch HN: Secoda (YC S21) – Searchable Company Data
- Scalable, fault-tolerant, low-latency storage service for real-time workloads
- Does Java have an open source package that can execute SQL on txt/csv?
- Avro SpecificRecord File Sink using apache flink is not compiling due to error incompatible types: FileSink<?> cannot be converted to SinkFunction<?>
- Would ParquetWriter from pyarrow automatically flush?
Index
What are some of the best open-source Big Data projects in Java? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Apache Flink | 17,595 |
| 2 | Presto | 12,873 |
| 3 | Apache Storm | 6,297 |
| 4 | Zeppelin | 5,486 |
| 5 | beam | 5,110 |
| 6 | Hazelcast | 4,631 |
| 7 | Trino | 4,460 |
| 8 | datahub | 4,122 |
| 9 | Apache Ignite | 4,009 |
| 10 | Apache Hive | 3,993 |
| 11 | vespa | 3,727 |
| 12 | Apache Calcite | 2,783 |
| 13 | Flume | 2,188 |
| 14 | Apache Drill | 1,616 |
| 15 | bookkeeper | 1,430 |
| 16 | Apache Parquet | 1,426 |
| 17 | DatumBox | 1,073 |
| 18 | dremio-oss | 989 |
| 19 | Hazelcast Jet | 951 |
| 20 | Apache Phoenix | 907 |
| 21 | Apache Accumulo | 895 |
| 22 | Rakam | 786 |
| 23 | Mockneat | 440 |
Are you hiring? Post a new remote job listing for free.
