Apache Spark Alternatives

Apache Spark reviews and mentions

• How to Install PySpark on Your Local Machine
  2 projects | dev.to | 9 Dec 2024

  If you’re stepping into the world of Big Data, you have likely heard of Apache Spark, a powerful distributed computing system. PySpark, the Python library for Apache Spark, is a favorite among data enthusiasts for its combination of speed, scalability, and ease of use. But setting it up on your local machine can feel a bit intimidating at first.

• How to Use PySpark for Machine Learning
  1 project | dev.to | 4 Dec 2024

  According to the Apache Spark official website, PySpark lets you utilize the combined strengths of ApacheSpark (simplicity, speed, scalability, versatility) and Python (rich ecosystem, matured libraries, simplicity) for “data engineering, data science, and machine learning on single-node machines or clusters.”

• Top FP technologies
  22 projects | dev.to | 29 Oct 2024

  spark

• Why Apache Spark RDD is immutable?
  1 project | dev.to | 29 Sep 2024

  Apache Spark is a powerful and widely used framework for distributed data processing, beloved for its efficiency and scalability. At the heart of Spark’s magic lies the RDD, an abstraction that’s more than just a mere data collection. In this blog post, we’ll explore why RDDs are immutable and the benefits this immutability provides in the context of Apache Spark.

• Spark SQL is getting pipe syntax
  1 project | news.ycombinator.com | 17 Sep 2024
• Intro to Ray on GKE
  3 projects | dev.to | 12 Sep 2024

  The Python Library components of Ray could be considered analogous to solutions like numpy, scipy, and pandas (which is most analogous to the Ray Data library specifically). As a framework and distributed computing solution, Ray could be used in place of a tool like Apache Spark or Python Dask. It’s also worthwhile to note that Ray Clusters can be used as a distributed computing solution within Kubernetes, as we’ve explored here, but Ray Clusters can also be created independent of Kubernetes.

• Avoid These Top 10 Mistakes When Using Apache Spark
  2 projects | dev.to | 28 Aug 2024

  We all know how easy it is to overlook small parts of our code, especially when we have powerful tools like Apache Spark to handle the heavy lifting. Spark's core engine is great at optimizing our messy, complex code into a sleek, efficient physical plan. But here's the catch: Spark isn't flawless. It's on a journey to perfection, sure, but it still has its limits. And Spark is upfront about those limitations, listing them out in the documentation (sometimes as little notes).

• IaaS vs PaaS vs SaaS: The Key Differences
  3 projects | dev.to | 18 Jul 2024

  One specific use case of the IaaS model is for deploying software that would have otherwise been bought as a SaaS. There are many such software from email servers to databases. You can choose to deploy MySQL in your infrastructure rather than buying from a MySQL SaaS provider. Other things you can deploy using the IaaS model include Mattermost for team collaboration, Apache Spark for data analytics, and SAP for Enterprise Resource Planning.

• How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop
  7 projects | dev.to | 17 Jun 2024

  In this project, I'm exploring the Medallion Architecture which is a data design pattern that organizes data into different layers based on structure and/or quality. I'm creating a fictional scenario where a large enterprise that has several branches across the country. Each branch receives purchase orders from an app and deliver the goods to their customers. The enterprise wants to identify the branch that receives the most purchase requests and the branch that has the minimum average delivery time. To achieve that, I've used Apache Spark as a distributed compute engine and Apache Hadoop, in particular HDFS, as my data storage layer. Apache Spark ingest, processes, and stores the app's data on HDFS to be served to a custom dashboard app. You can find all about it, in this Github repo

• Shades of Open Source - Understanding The Many Meanings of "Open"
  9 projects | dev.to | 15 Jun 2024

  In contrast, Databricks maintains internal forks of Spark, Delta Lake, and Unity Catalog, using the same names for both the open-source versions and the features specific to the Databricks platform. While they do provide separate documentation, online discussions often reflect confusion about how to use features in the open-source versions that only exist on the Databricks platform. This creates a "muddying of the waters" between what is open and what is proprietary. This isn't an issue if you are a Databricks user, but it can be quite confusing for those who want to use these tools outside of the Databricks ecosystem.

• A note from our sponsor - CodeRabbit
  coderabbit.ai | 12 Dec 2024

  Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR. Learn more →