Python apache-spark

Open-source Python projects categorized as apache-spark

Top 12 Python apache-spark Projects

  • MLflow

    Open source platform for the machine learning lifecycle

  • Project mention: Exploring Open-Source Alternatives to Landing AI for Robust MLOps | | 2023-12-13

    Platforms such as MLflow monitor the development stages of machine learning models. In parallel, Data Version Control (DVC) brings version control system-like functions to the realm of data sets and models.

  • flintrock

    A command-line tool for launching Apache Spark clusters.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
  • quinn

    pyspark methods to enhance developer productivity 📣 👯 🎉 (by MrPowers)

  • PySpark-Boilerplate

    A boilerplate for writing PySpark Jobs

  • sparktorch

    Train and run Pytorch models on Apache Spark.

  • dataproc-templates

    Dataproc templates and pipelines for solving simple in-cloud data tasks

  • Apache-Spark-Guide

    Apache Spark Guide

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • covid-19-data-engineering-pipeline

    A Covid-19 data pipeline on AWS featuring PySpark/Glue, Docker, Great Expectations, Airflow, and Redshift, templated in CloudFormation and CDK, deployable via Github Actions.

  • Traffic-Data-Analysis-with-Apache-Spark-Based-on-Mobile-Robot-Data

    Mobile robot data were analyzed with Apache-Spark to extract five different statistical result such as travel time, waiting time, average speed, occupancy and density were produced.

  • xonai-dashboard

    A Grafana-based application to assist Big Data infrastructure optimization initiatives where Spark applications are a dominant cost driver

  • Project mention: Show HN: Open sourcing a Big Data monitoring tool | | 2024-03-29
  • livyc

    Apache Spark as a Service with Apache Livy Client

  • transactional-datalake-using-amazon-msk-and-apache-iceberg-on-aws-glue

    Stream CDC into an Amazon S3 data lake in Apache Iceberg format with AWS Glue Streaming using Amazon MSK and MSK Connect (Debezium)

  • Project mention: Writing simple Python scripts faster with Amazon Q | | 2024-01-11

    transactional-datalake-using-amazon-msk-serverless-and-apache-iceberg-on-aws-glue 2024-01-10T01:26:56Z aws-msk-serverless-cdc-data-pipeline-with-debezium 2024-01-09T01:03:38Z aws-healthlake-smart-on-fhir 2024-01-08T23:05:17Z aws-greengrass-custom-components 2024-01-08T11:34:12Z graviton-developer-workshop 2024-01-08T03:30:31Z msk-flink-streaming-cdk 2024-01-08T02:25:39Z rag-with-amazon-postgresql-using-pgvector 2024-01-06T04:47:41Z queueTransfer_ContactTraceRecordSupport-for-Service-Cloud-Voice 2024-01-05T20:34:14Z amazon-chime-sdk-voice-voice-translator 2024-01-05T17:25:54Z private-s3-vpce 2024-01-05T06:38:52Z bedrock-contact-center-tasks-eval 2024-01-04T21:46:51Z clickstream-sdk-samples 2024-01-04T07:21:52Z aws-msk-cdc-data-pipeline-with-debezium 2024-01-04T04:09:22Z transactional-datalake-using-amazon-msk-and-apache-iceberg-on-aws-glue 2024-01-04T03:39:04Z ..

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-03-29.

Python apache-spark related posts


What are some of the best open-source apache-spark projects in Python? This list will help you:

Project Stars
1 MLflow 17,185
2 flintrock 630
3 quinn 573
4 PySpark-Boilerplate 390
5 sparktorch 334
6 dataproc-templates 109
7 Apache-Spark-Guide 26
8 covid-19-data-engineering-pipeline 22
9 Traffic-Data-Analysis-with-Apache-Spark-Based-on-Mobile-Robot-Data 10
10 xonai-dashboard 10
11 livyc 3
12 transactional-datalake-using-amazon-msk-and-apache-iceberg-on-aws-glue 1
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives