|4 days ago||about 14 hours ago|
|Apache License 2.0||Apache License 2.0|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Visualization using Pyspark Dataframe
2 projects | reddit.com/r/dataengineering | 14 May 2022
Have you tried Apache Zepellin I remember that you can pretty print spark dataframes directly on it with z.show(df)
Fast CSV Processing with SIMD
5 projects | news.ycombinator.com | 4 Dec 2021
I used to use Zeppelin, some kind of Jupyter Notebook for Spark (that supports Parquet). But it may be better alternatives.
What libraries do you use for machine learning and data visualizing in scala?
5 projects | reddit.com/r/scala | 27 Nov 2021
Another more widely used notebooks for scala and spark: https://zeppelin.apache.org/
How to use IPython in Apache Zeppelin Notebook
2 projects | dev.to | 10 Jul 2021
 Apache Zeppelin http://zeppelin.apache.org/  Zeppelin notebooks website http://zeppelin-notebook.com/.  Zeppelin notebooks git repo https://github.com/zjffdu/zeppelin-notebook
BI Application in Golang.
2 projects | reddit.com/r/golang | 7 May 2021
Using InterSystems Caché and Apache Zeppelin
1 project | dev.to | 25 Apr 2021
For all who think: What the heck is Apache Zeppelin? Here are some details what the project site says:
Is there a way to collaborate in real-time for Jupyter Notebooks?
2 projects | reddit.com/r/learnpython | 21 Mar 2021
Check out Zeppelin. It's similar to Jupyter and allows real-time editing by multiple users. https://zeppelin.apache.org/
Introduce Cache Hints to Spark SQL
1 project | reddit.com/r/apachespark | 8 Aug 2022
I've submitted PR-37355 to introduce cache hints to Spark SQL. This feature works as a supplement for CACHE/UNCACHE commands, pure SQL users can operate the cache in their queries directly without extra definitions. Also, cache skipping is supported by hints.
Late Night Random Discussion Thread - 08 August, 2022
1 project | reddit.com/r/indiasocial | 8 Aug 2022
ab thik hai?
Efficiently processing large amounts of data that needs to be grouped by multiple parameters
2 projects | reddit.com/r/javahelp | 20 Jul 2022
Naturally it's difficult to suggest tools, but something like https://spark.apache.org or https://jet-start.sh ?
is anyone want to join maintaining spark java framework?
2 projects | reddit.com/r/java | 21 Jun 2022
Wow, this has nothing to do with Apache Spark (https://spark.apache.org/), the wildly popular JVM based data processing framework.
How-to-Guide: Contributing to Open Source
19 projects | reddit.com/r/dataengineering | 11 Jun 2022
Perform computation over 500 million vectors
1 project | reddit.com/r/bigdata | 8 Jun 2022
I would guess that Apache Spark would be an okay choice. With data stored locally in avro or parquet files. Just processing the data in python would also work, IMO.
DeWitt Clause, or Can You Benchmark %DATABASE% and Get Away With It
21 projects | dev.to | 2 Jun 2022
Apache Drill, Druid, Flink, Hive, Kafka, Spark
Optimizing Distributed Joins: The Case of Google Cloud Spanner and DataStax Astra DB
3 projects | dev.to | 31 May 2022
Shuffle and broadcast joins are more suitable for batch or near real-time analytics. For example, they are used in Apache Spark as the main join strategies. Co-located and pre-computed joins are faster and can be used for online transaction processing with real-time applications. They frequently rely on organizing data based on unique storage schemes supported by a database.
What do I need to know about distributed algorithms and systems?
1 project | reddit.com/r/AskProgramming | 22 May 2022
You generally want to keep your data in memory, rather than disk, to keep things reasonably fast. A system like Apache Spark tries to do this for you, spilling to disk when needed. In general, I'd recommend researching Spark, since it will cover a lot of the concepts you care about.
How to use Spark and Pandas to prepare big data
3 projects | dev.to | 10 May 2022
Apache Spark is one of the most actively developed open-source projects in big data. The following code examples require that you have Spark set up and can execute Python code using the PySpark library. The examples also require that you have your data in Amazon S3 (Simple Storage Service). All this is set up on AWS EMR (Elastic MapReduce).
What are some alternatives?
Trino - Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Pytorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration
Scalding - A Scala API for Cascading
mrjob - Run MapReduce jobs on Hadoop or Amazon Web Services
luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Smile - Statistical Machine Intelligence & Learning Engine
Apache Arrow - Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
Apache Calcite - Apache Calcite
Scio - A Scala API for Apache Beam and Google Cloud Dataflow.
Apache Flink - Apache Flink