ci-cd-serverless-spark
Demo for GitHub Universe 2022 (by dacort)
pyspark-testing-env
Example Repo to have full end to end pyspark testing via docker-compose (by emmc15)
| ci-cd-serverless-spark | pyspark-testing-env | |
|---|---|---|
| 2 | 3 | |
| 11 | 28 | |
| - | - | |
| 10.0 | 10.0 | |
| over 1 year ago | over 1 year ago | |
| Python | Python | |
| - | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ci-cd-serverless-spark
Posts with mentions or reviews of ci-cd-serverless-spark.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-03-14.
-
CI/CD for AWS Glue jobs ?
My second example is more recent but is for EMR Serverless. It might still be useful, though - it uses GitHub Actions to build/deploy versioned artifacts to s3. The full code is in this ci-cd-serverless-spark repo.
-
End to End Pyspark Testing CI/CD Example Repo
I’ve also got a similar repo for full CI/CD with EMR Serverless.
pyspark-testing-env
Posts with mentions or reviews of pyspark-testing-env.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-02-05.
-
Docker question when building a data engineering stack
If you're looking for something that's more solely on the developer experience side rather than production example, I have a base repo here. It has:
- End-to-End Pyspark and S3 Docker Compose Repo Setup
- End to End Pyspark Testing CI/CD Example Repo
What are some alternatives?
When comparing ci-cd-serverless-spark and pyspark-testing-env you can also consider the following projects:
athena-glue-service-logs - Glue scripts for converting AWS Service Logs for use in Athena
spark-local-environment - An example of using EMR Serverless container image for local environment