Apache Flink VS Smile

7. Apache Flink | Github | tutorial

Does anyone have fully running Pyflink code snippet to read from Kafka using the new Flink DataStream (KafkaSource) API and just print out the output to console or write it out to a file. Most of the examples and the official Flink GitHubare using the old API (FlinkKafkaConsumer).

I'm trying to use https://github.com/mauricioaniche/ck to analyze the ck metrics of https://github.com/apache/flink. I have the latest version of java downloaded and I have the latest version of apache maven downloaded too. My environment variables are set correctly. I'm in the correct directory as well. However, when I run mvn clean compile package in powershell it always says build error. I've tried looking up the errors but there's so many. https://imgur.com/a/Zk8Snsa I'm very new to programming in general so any suggestions would be appreciated.

Looking at the project on github, it seems like they should have a pom in the root dir https://github.com/apache/flink/blob/master/pom.xml

Akka is used only as a possible RPC implementation, isn't it?

Apache Drill, Druid, Flink, Hive, Kafka, Spark

It took me some time to get a good grasp of the power of SQL; and it really kicked in when I learned about optimization rules. It's a program that you rewrite, just like an optimizing compiler would.

You state what you want; you have different ways to fetch and match and massage data; and you can search through this space to produce a physical plan. Hopefully you used knowledge to weight parts to be optimized (table statistics, like Java's JIT would detect hot spots).

I find it fascinating to peer through database code to see what is going on. Lately, there's been new advances towards streaming databases, which bring a whole new design space. For example, now you have latency of individual new rows to optimize for, as opposed to batch it whole to optimize the latency of a dataset. Batch scanning will be benefit from better use of your CPU caches.

And maybe you could have a hybrid system which reads history from a log and aggregates in a batched manner, and then switches to another execution plan when it reaches the end of the log.

If you want to have a peek at that here are Flink's set of rules [1], generic and stream-specific ones. The names can be cryptic, but usually give a good sense of what is going on. For example: PushFilterIntoTableSourceScanRule makes the WHERE clause apply the earliest possible, to save some CPU/network bandwidth further down. PushPartitionIntoTableSourceScanRule tries to make a fan-out/shuffle happen the earliest possible, so that parallelism can be made use of.

[1] https://github.com/apache/flink/blob/5f8fb304fb5d68cdb0b3e3c...

[1]: https://mvnrepository.com/artifact/org.apache.avro/avro-maven-plugin/1.8.2 [2]: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java [3]: https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/datastream/file_sink/ [4]: https://github.com/apache/flink/blob/c81b831d5fe08d328251d91f4f255b1508a9feb4/flink-end-to-end-tests/flink-file-sink-test/src/main/java/FileSinkProgram.java [5]: https://github.com/rajcspsg/streaming-file-sink-demo

> I don't think it's right to recommend that new users move away from the package because of licensing issues

I was going to chime in to agree but then I saw how this was done - a completely innocuous looking commit:

https://github.com/haifengl/smile/commit/6f22097b233a3436519...

And literally no mention in the release notes:

https://github.com/haifengl/smile/releases/tag/v3.0.0

I think if you are going to change license especially in a way that makes it less permissive you need to be super open and clear about both the fact you are doing it and your reasons for that. This is done so silently as to look like it is intentionally trying to mislead and trick people.

So maybe I wouldn't say to move away because of the specific license, but it's legitimate to avoid something when it's so clearly driven by a single entity and that entity acts in a way that isn't trustworthy.

Check out Smile too.

Although it may be a bit more work, you can do both machine learning and AI in Java. If you are doing deep learning, you can use DeepJavaLibrary (I do work on this one at Amazon). If you are looking for other ML algorithms, I have seen Smile, Tribuo, or some around Spark.

For deploying a trained model there are a bunch of options that use Java on top of some native runtime like TF-Java (which I co-lead), ONNX Runtime, pytorch has inference for TorchScript models. Training deep learning models is harder, though you can do it for some of them in DJL. Training more standard ML models is much simpler, either via Tribuo, or using things like LibSVM & XGBoost directly, or other libraries like SMILE or WEKA.

I use smile https://github.com/haifengl/smile with ammonite and it feels pretty easy/good to work with. Of course for pure looking at data, and exploration, you're not going to beat python.

Actually, it does. Scala has Spark for data science and some ML libs like Smile.

I guess I must have a mistake somewhere. It's not much code. it's written in Kotlin with smile. My dataset is only about 32MB. I load the dataset into memory. I then use 80% of the data for training, and the other for later testing. I get just the columns I need and store them in the variable dataset.

I've heard good things about Smile, probably beats libs like Weka by far. I'm not sure if you can load a scikit-learn model though, so you might need to retrain the model in Kotlin.

I was using Smile for some period - https://haifengl.github.io/ - it's quite small and lightweight Java lib with some very basic algorithms - I was using in particularly cauterization. Along with this it provides Scala API.