Java Spark

Open-source Java projects categorized as Spark

Top 23 Java Spark Projects

  • Deeplearning4j

    Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation.

  • doris

    Apache Doris is an easy-to-use, high performance and unified analytics database.

    Project mention: Five Apache projects you probably didn't know about | | 2023-12-21

    Apache Doris is a real-time data warehouse.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • Alluxio (formerly Tachyon)

    Alluxio, data orchestration for analytics and machine learning in the cloud

  • Zeppelin

    Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

    Project mention: Serverless Apache Zeppelin on AWS | | 2024-02-04

    Now we can proceed with the definition of Apache Zeppelin. It is a web-based notebook that enables data-driven, interactive data analytics and collaborative documents with Python, Scala, SQL, Spark, and more. You can execute code and even schedule a job (via cron) to run at regular intervals.

  • RoaringBitmap

    A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Tablesaw, and many others

    Project mention: Iterating over Bit Sets Quickly | | 2024-02-24

    I was recently reading about Roaring which is a highly optimized compressed bitset implementation. I reccomend reading about it if you are interested in this sort of thing. The talk at is especially good.

  • linkis

    Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

  • LakeSoul

    LakeSoul is an end-to-end, realtime and cloud native Lakehouse framework with fast data ingestion, concurrent update and incremental data analytics on cloud storages for both BI and AI applications.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • elassandra

    Elassandra = Elasticsearch + Apache Cassandra

  • incubator-paimon

    Apache Paimon(incubating) is a streaming data lake platform that supports high-speed data ingestion, change data tracking and efficient real-time analytics.

    Project mention: Open Source Advent Fun Wraps Up! | | 2024-01-05

    18. Apache Paimon | Github | tutorial

  • kylo

    Kylo is a data lake management software platform and framework for enabling scalable enterprise-class data lakes on big data technologies such as Teradata, Apache Spark and/or Hadoop. Kylo is licensed under Apache 2.0. Contributed by Teradata Inc.

    Project mention: GitHub – GSA/code-gov: An informative repo for all repos | | 2023-09-09 :

    > You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.*

    > [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you

    > To load a Parquet file, pass a URL to `?parquet=`

    > [...]*

    There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet.

    datasette plugins are written in Python and/or JS w/ pluggy:

  • zingg

    Scalable identity resolution, entity resolution, data mastering and deduplication using ML

  • nessie

    Nessie: Transactional Catalog for Data Lakes with Git-like semantics

    Project mention: A deep dive into the concept and world of Apache Iceberg Catalogs | | 2024-03-01

    Nessie is an innovative open-source catalog that extends beyond the traditional catalog capabilities in the Apache Iceberg ecosystem, introducing git-like features to data management. This catalog not only tracks table metadata but also allows users to capture commits at a holistic level, enabling advanced operations such as multi-table transactions, rollbacks, branching, and tagging. These features provide a new layer of flexibility and control over data changes, resembling version control systems in software development.

  • Sparkler

    Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

  • spark-bigquery-connector

    BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

  • incubator-uniffle

    Uniffle is a high performance, general purpose Remote Shuffle Service.

    Project mention: Apache Uniffle: a high performance remote shuffle service for Spark | | 2023-10-02
  • dataCompare

    big data comparison and data profiling platform: low code,data comparison and data profiling

  • rumble

    ⛈️ RumbleDB 1.21.0 "Hawthorn blossom" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more (by RumbleDB)

  • incubator-wayang

    Apache Wayang(incubating) is the first cross-platform data processing system.

    Project mention: Support different jdbc platforms and multiple instances of same DBMS | /r/ApacheWayang | 2023-12-05
  • batch-processing-gateway

    The gateway component to make Spark on K8s much easier for Spark users.

  • big-data-pipeline-lambda-arch

    A full big data pipeline (Lambda Architecture) with Spark, Kafka, HDFS and Cassandra.

  • hadoopcryptoledger

    Hadoop Crypto Ledger - Analyzing CryptoLedgers, such as Bitcoin Blockchain, on Big Data platforms, such as Hadoop/Spark/Flink/Hive

  • lighter

    REST API for Apache Spark on K8S or YARN

  • squashql

    Official repository of SquashQL, the SQL query engine for multi-dimensional and hierarchical analysis that empowers your SQL database

    Project mention: Show HN: SQL Polyglot | | 2023-12-16

    I am building a SQL like query engine with a typescript query builder. It aims to provide multidimensional query capabilities (think about olap cube). The query engine is compatible with several databases (sucks, clickhouse, bigqueryx, snowflake...). Our users are developers building decision making applications. They are using for instance duckdb to develop locally their application and use a cloud provider in production. Here's the link Feel free to ask me any questions.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-03-01.

Java Spark related posts


What are some of the best open-source Spark projects in Java? This list will help you:

Project Stars
1 Deeplearning4j 13,367
2 doris 10,915
3 Alluxio (formerly Tachyon) 6,581
4 Zeppelin 6,236
5 RoaringBitmap 3,335
6 linkis 3,212
7 LakeSoul 2,268
8 elassandra 1,703
9 incubator-paimon 1,703
10 kylo 1,086
11 zingg 860
12 nessie 789
13 Sparkler 409
14 spark-bigquery-connector 340
15 incubator-uniffle 340
16 dataCompare 228
17 rumble 205
18 incubator-wayang 163
19 batch-processing-gateway 162
20 big-data-pipeline-lambda-arch 161
21 hadoopcryptoledger 142
22 lighter 75
23 squashql 39
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives