DataFrames.jl VS ScikitLearn.jl

I don't think we've seen the final state of it though. v1.9 really gives people the proper tools for solving latency problems. Before, invalidations hurt a little, but it was also kind of a wash because LLVM code didn't precompile, so you could spend time setting up a Snoopprecompile and fix some invalidations and end up LLVM bound saving 1 second out of 20. But with package images it's almost always better to fix precompilation. The only thing hampering time much now is the `using` time went up, but as mentioned in the Reddit post there's a lot of ideas for what to do there. The other thing is package extensions, which cut down the amount of code to load. There's tons of PRs floating around the ecosystem turning things into extensions, and thus cutting down the overall code that is actually ran and loaded.

This means that in a few months, people will start to see some major tangible benefits from following the compilation improvement practices laid out here and https://sciml.ai/news/2022/09/21/compile_time/. I think then you'll have a lot more people start to take all of these new tools seriously and it will be standard to incorporate them into packages. Right now they are still kind of niche things for packages with known TTFX problems, but I think come v1.9 you'll see every major package use all of these methods.

> Different packages are affected differently

I think this is one of the pieces that's effected by this. I don't think "Julia has become optimised for running Plots.jl" is quite correct. Julia's compilation and runtime is much more optimized for well-inferred code, which Plots.jl is not. However, the compiler developers have been using Plots.jl as a test case for all of these new tools, and therefore its SnoopPrecompile and invalidations have gotten some dramatic improvements because those required ecosystem changes I mentioned are being done by the compiler team for this specific package. Plots.jl had things like precompilation snooping way back before there was even a package for it, the earliest I know of was around v1.0. Meanwhile, DataFrames.jl only setup its precompilation snooping 7 months ago (https://github.com/JuliaData/DataFrames.jl/pull/3182), which I would presume was just in time for the v1.8 mark on your plot and is one of the big reasons for having a sudden drop (which continues into v1.9 because of package images).

What I mean to say then is that, I think all packages will get the improvements we've seen from Plots.jl, but package authors will need to update their packages in order for that to happen. Some packages have already done this, many have not.

IJulia also supports viewing and manipulating tables. To create a table, first install the DataFrames.jl package by running the following command in a new cell:

It were just a few percents of all possible manipulations that you can do with data using DataFrames.jl library. Read more about it in the documentation.

DataFrames.jl

DataFrames.jl and XLSX.jl for JSON, CSV, and XLSX files

Julia also has the CSV.jl library for reading/writing csv files, the DataFrames.jl library for manipulating data like pandas, and Images.jl for image processing/analysis. However, since Julia is so much newer than Python, the Julia libraries are almost never as feature rich as their Python counterparts.

Looks like it's single threaded according to this open issue: https://github.com/JuliaData/DataFrames.jl/issues/2626

For machine learning, we will use SciKitLearn.jl library, which replicates SciKit-Learn library for Python. It provides an interface for commonly used machine learning models like Logistic Regression, Decission Tree or Random Forest. SciKitLearn.jl is not a single package but a rich ecosystem with many packages, and you need to select which of them to install and import. You can find a list of supported models here. Some of them are built-in Julia models, others are imported from Python. Also, the SciKitLearn.jl has a lot of tools to tune the learning process and evaluate results.

MLJ.jl is a good Julia ML framework. There's also a Scikitlearn.jl but its more of a wrapper around the sklearn I believe

That repo appears to be deprecated. Since you called using ScikitLearn, I imagine you should check this repo instead: https://github.com/cstjean/ScikitLearn.jl

Imho, Julia is more diverse in the sense that there is not a single popular ML library. Maybe the Julian equivalent for scikit-learn is MLJ.jl. There is also ScikitLearn.jl, which defines the usual interface of scikit-learn models, and specific algorithms then implement this interface.