Apache Spark
mrjob
Our great sponsors
| Apache Spark | mrjob | |
|---|---|---|
| 48 | 1 | |
| 32,903 | 2,579 | |
| 1.6% | 0.1% | |
| 10.0 | 0.0 | |
| 3 days ago | 2 months ago | |
| Scala | Python | |
| Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Apache Spark
-
What do I need to know about distributed algorithms and systems?
You generally want to keep your data in memory, rather than disk, to keep things reasonably fast. A system like Apache Spark tries to do this for you, spilling to disk when needed. In general, I'd recommend researching Spark, since it will cover a lot of the concepts you care about.
-
How to use Spark and Pandas to prepare big data
Apache Spark is one of the most actively developed open-source projects in big data. The following code examples require that you have Spark set up and can execute Python code using the PySpark library. The examples also require that you have your data in Amazon S3 (Simple Storage Service). All this is set up on AWS EMR (Elastic MapReduce).
-
AWS Glue: what is it and how does it work?
With Glue, Apache Spark runs in the background. But if this is the first time you’ve heard of the popular open-source analytics engine, it may take you a while to familiarize yourself with the cloud software.
-
Real-time Open Source Indexes: Databases, Headless CMSs and Static Site Generators
Spark SQL (302 active contributors).
-
Top Responsibilities of a Data Engineering Manager
What’s more, picking the right technology is always evolving. New tools come out all the time, often with different functionality than existing tools. So it’s important that you stay up-to-date on what technologies are available and their latest features. For example, four years ago Apache Spark was completely unknown but today it is quickly becoming the de facto standard for stream processing.
-
Apache Spark, Hive, and Spring Boot — Testing Guide
In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.
-
Cannot find col function in pyspark
from pyspark.sql.functions import col but when I try to look it up in the Github source code I find no col function in functions.py file, how can python import a function that doesn't exist?
-
How To Start Your Next Data Engineering Project
Apache Spark
-
Big Data Processing, EMR with Spark and Hadoop | Python, PySpark
Apache Spark is an open-source, distributed processing system used for big data workloads. Wanna dig more dipper?
-
Rule
Spark Legend-er
mrjob
What are some alternatives?
Trino - Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
Scalding - A Scala API for Cascading
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Pytorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration
Smile - Statistical Machine Intelligence & Learning Engine
Weka
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Apache Arrow - Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
Apache Calcite - Apache Calcite
Scio - A Scala API for Apache Beam and Google Cloud Dataflow.
Deeplearning4j - Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation.