Zeppelin VS Apache Spark

Have you tried Apache Zepellin I remember that you can pretty print spark dataframes directly on it with z.show(df)

I used to use Zeppelin, some kind of Jupyter Notebook for Spark (that supports Parquet). But it may be better alternatives.

https://zeppelin.apache.org/

Another more widely used notebooks for scala and spark: https://zeppelin.apache.org/

[1] Apache Zeppelin http://zeppelin.apache.org/ [2] Zeppelin notebooks website http://zeppelin-notebook.com/. [3] Zeppelin notebooks git repo https://github.com/zjffdu/zeppelin-notebook

Apache Zeppelin

For all who think: What the heck is Apache Zeppelin? Here are some details what the project site says:

Check out Zeppelin. It's similar to Jupyter and allows real-time editing by multiple users. https://zeppelin.apache.org/

I've submitted PR-37355 to introduce cache hints to Spark SQL. This feature works as a supplement for CACHE/UNCACHE commands, pure SQL users can operate the cache in their queries directly without extra definitions. Also, cache skipping is supported by hints.

ab thik hai?

Naturally it's difficult to suggest tools, but something like https://spark.apache.org or https://jet-start.sh ?

Wow, this has nothing to do with Apache Spark (https://spark.apache.org/), the wildly popular JVM based data processing framework.

Apache Spark

I would guess that Apache Spark would be an okay choice. With data stored locally in avro or parquet files. Just processing the data in python would also work, IMO.

Apache Drill, Druid, Flink, Hive, Kafka, Spark

Shuffle and broadcast joins are more suitable for batch or near real-time analytics. For example, they are used in Apache Spark as the main join strategies. Co-located and pre-computed joins are faster and can be used for online transaction processing with real-time applications. They frequently rely on organizing data based on unique storage schemes supported by a database.

You generally want to keep your data in memory, rather than disk, to keep things reasonably fast. A system like Apache Spark tries to do this for you, spilling to disk when needed. In general, I'd recommend researching Spark, since it will cover a lot of the concepts you care about.

Apache Spark is one of the most actively developed open-source projects in big data. The following code examples require that you have Spark set up and can execute Python code using the PySpark library. The examples also require that you have your data in Amazon S3 (Simple Storage Service). All this is set up on AWS EMR (Elastic MapReduce).