|Apache Mahout||Apache Spark|
|about 2 months ago||3 days ago|
|Apache License 2.0||Apache License 2.0|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
We haven't tracked posts mentioning Apache Mahout yet.
Tracking mentions began in Dec 2020.
Introduce Cache Hints to Spark SQL
1 project | reddit.com/r/apachespark | 8 Aug 2022
I've submitted PR-37355 to introduce cache hints to Spark SQL. This feature works as a supplement for CACHE/UNCACHE commands, pure SQL users can operate the cache in their queries directly without extra definitions. Also, cache skipping is supported by hints.
Late Night Random Discussion Thread - 08 August, 2022
1 project | reddit.com/r/indiasocial | 8 Aug 2022
ab thik hai?
Efficiently processing large amounts of data that needs to be grouped by multiple parameters
2 projects | reddit.com/r/javahelp | 20 Jul 2022
Naturally it's difficult to suggest tools, but something like https://spark.apache.org or https://jet-start.sh ?
is anyone want to join maintaining spark java framework?
2 projects | reddit.com/r/java | 21 Jun 2022
Wow, this has nothing to do with Apache Spark (https://spark.apache.org/), the wildly popular JVM based data processing framework.
How-to-Guide: Contributing to Open Source
19 projects | reddit.com/r/dataengineering | 11 Jun 2022
Perform computation over 500 million vectors
1 project | reddit.com/r/bigdata | 8 Jun 2022
I would guess that Apache Spark would be an okay choice. With data stored locally in avro or parquet files. Just processing the data in python would also work, IMO.
DeWitt Clause, or Can You Benchmark %DATABASE% and Get Away With It
21 projects | dev.to | 2 Jun 2022
Apache Drill, Druid, Flink, Hive, Kafka, Spark
Optimizing Distributed Joins: The Case of Google Cloud Spanner and DataStax Astra DB
3 projects | dev.to | 31 May 2022
Shuffle and broadcast joins are more suitable for batch or near real-time analytics. For example, they are used in Apache Spark as the main join strategies. Co-located and pre-computed joins are faster and can be used for online transaction processing with real-time applications. They frequently rely on organizing data based on unique storage schemes supported by a database.
What do I need to know about distributed algorithms and systems?
1 project | reddit.com/r/AskProgramming | 22 May 2022
You generally want to keep your data in memory, rather than disk, to keep things reasonably fast. A system like Apache Spark tries to do this for you, spilling to disk when needed. In general, I'd recommend researching Spark, since it will cover a lot of the concepts you care about.
How to use Spark and Pandas to prepare big data
3 projects | dev.to | 10 May 2022
Apache Spark is one of the most actively developed open-source projects in big data. The following code examples require that you have Spark set up and can execute Python code using the PySpark library. The examples also require that you have your data in Amazon S3 (Simple Storage Service). All this is set up on AWS EMR (Elastic MapReduce).
What are some alternatives?
Trino - Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Pytorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration
Scalding - A Scala API for Cascading
mrjob - Run MapReduce jobs on Hadoop or Amazon Web Services
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Smile - Statistical Machine Intelligence & Learning Engine
Apache Arrow - Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
Apache Calcite - Apache Calcite
Scio - A Scala API for Apache Beam and Google Cloud Dataflow.
Deeplearning4j - Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learning using automatic differentiation.