Apache Spark VS Smile

You generally want to keep your data in memory, rather than disk, to keep things reasonably fast. A system like Apache Spark tries to do this for you, spilling to disk when needed. In general, I'd recommend researching Spark, since it will cover a lot of the concepts you care about.

Apache Spark is one of the most actively developed open-source projects in big data. The following code examples require that you have Spark set up and can execute Python code using the PySpark library. The examples also require that you have your data in Amazon S3 (Simple Storage Service). All this is set up on AWS EMR (Elastic MapReduce).

With Glue, Apache Spark runs in the background. But if this is the first time you’ve heard of the popular open-source analytics engine, it may take you a while to familiarize yourself with the cloud software.

Spark SQL (302 active contributors).

What’s more, picking the right technology is always evolving. New tools come out all the time, often with different functionality than existing tools. So it’s important that you stay up-to-date on what technologies are available and their latest features. For example, four years ago Apache Spark was completely unknown but today it is quickly becoming the de facto standard for stream processing.

In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

from pyspark.sql.functions import col but when I try to look it up in the Github source code I find no col function in functions.py file, how can python import a function that doesn't exist?

Apache Spark

Apache Spark is an open-source, distributed processing system used for big data workloads. Wanna dig more dipper?

Spark Legend-er

I use smile https://github.com/haifengl/smile with ammonite and it feels pretty easy/good to work with. Of course for pure looking at data, and exploration, you're not going to beat python.

Actually, it does. Scala has Spark for data science and some ML libs like Smile.

I guess I must have a mistake somewhere. It's not much code. it's written in Kotlin with smile. My dataset is only about 32MB. I load the dataset into memory. I then use 80% of the data for training, and the other for later testing. I get just the columns I need and store them in the variable dataset.

I've heard good things about Smile, probably beats libs like Weka by far. I'm not sure if you can load a scikit-learn model though, so you might need to retrain the model in Kotlin.

I was using Smile for some period - https://haifengl.github.io/ - it's quite small and lightweight Java lib with some very basic algorithms - I was using in particularly cauterization. Along with this it provides Scala API.