pyspark-on-aws-emr
demo-code
| pyspark-on-aws-emr | demo-code | |
|---|---|---|
| 1 | 2 | |
| 24 | 28 | |
| - | - | |
| 3.6 | 0.0 | |
| almost 2 years ago | 3 months ago | |
| Python | Jupyter Notebook | |
| Apache License 2.0 | Creative Commons Zero v1.0 Universal |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pyspark-on-aws-emr
-
Data Engineering Projects for Beginners
Building Big Data Pipelines in the Cloud with AWS EMR
demo-code
-
Difference between databricks and EMR +jupyper?
Sounds good - the setup steps are still a bit manual since it’s in preview, but CloudFormation support was just released and I’ll be posting a template in my demo-code repo next week.
-
Lessons From Building a Highly Reproducible Data Warehouse
Pretty nice article! I've been experimenting lately with using AWS CDK to deploy a big data stack including EMR on EKS and Airflow. The code is available on my GitHub - dacort/demo-code and I also just recorded a video stepping through how it's all put together.
What are some alternatives?
docker-livy - Dockerizing and Consuming an Apache Livy environment
quilt - Quilt is a data mesh for connecting people with actionable data
TypedPyspark - Type-annotate your spark dataframes and validate them
aws-data-wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL). [Moved to: https://github.com/aws/aws-sdk-pandas]
Traffic-Data-Analysis-with-Apache-Spark-Based-on-Mobile-Robot-Data - Mobile robot data were analyzed with Apache-Spark to extract five different statistical result such as travel time, waiting time, average speed, occupancy and density were produced.
amazon-emr-with-delta-lake - Amazon EMR Notebook to show how to read from and write to Delta tables with Amazon EMR
livyc - Apache Spark as a Service with Apache Livy Client
AWS Data Wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
IDEA - Text Data Visualizer with Django
cc-notebooks - Various Jupyter notebooks about Common Crawl data
pyspark-starter - Starter pyspark code with a working combination of all versions
reactivesearch - Search UI components for React and Vue