Zeppelin VS DataProfiler

Now we can proceed with the definition of Apache Zeppelin. It is a web-based notebook that enables data-driven, interactive data analytics and collaborative documents with Python, Scala, SQL, Spark, and more. You can execute code and even schedule a job (via cron) to run at regular intervals.

Have you tried Apache Zepellin I remember that you can pretty print spark dataframes directly on it with z.show(df)

I used to use Zeppelin, some kind of Jupyter Notebook for Spark (that supports Parquet). But it may be better alternatives.

https://zeppelin.apache.org/

Another more widely used notebooks for scala and spark: https://zeppelin.apache.org/

[1] Apache Zeppelin http://zeppelin.apache.org/ [2] Zeppelin notebooks website http://zeppelin-notebook.com/. [3] Zeppelin notebooks git repo https://github.com/zjffdu/zeppelin-notebook

Apache Zeppelin

For all who think: What the heck is Apache Zeppelin? Here are some details what the project site says:

Check out Zeppelin. It's similar to Jupyter and allows real-time editing by multiple users. https://zeppelin.apache.org/

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler

For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose