db-benchmark VS FromFile.jl

Real-world performance is complicated since data science covers a lot of use cases.

If you're just reading a small CSV to do analysis on it, then there will be no human-perceptible difference between Polars and Pandas. If you're reading a larger CSV with 100k rows, there still won't be much of a perceptible difference.

Per this (old) benchmark, there are differences once you get into 500MB+ territory: https://h2oai.github.io/db-benchmark/

I do think it was important for duckdb to put out a new version of the results as the earlier version of that benchmark [1] went dormant with a very old version of duckdb with very bad performance, especially against polars.

[1] https://h2oai.github.io/db-benchmark/

https://news.ycombinator.com/item?id=33270638 :

> Apache Ballista and Polars do Apache Arrow and SIMD.

> The Polars homepage links to the "Database-like ops benchmark" of {Polars, data.table, DataFrames.jl, ClickHouse, cuDF, spark, (py)datatable, dplyr, pandas, dask, Arrow, DuckDB, Modin,} but not yet PostgresML? https://h2oai.github.io/db-benchmark/ *

LLM -> Vector database: https://en.wikipedia.org/wiki/Vector_database

/? inurl:awesome site:github.com "vector database"

I agree with your conclusion but want to add that switching from Julia may not make sense either.

According to these benchmarks: https://h2oai.github.io/db-benchmark/, DF.jl is the fastest library for some things, data.table for others, polars for others. Which is fastest depends on the query and whether it takes advantage of the features/properties of each.

For what it's worth, data.table is my favourite to use and I believe it has the nicest ergonomics of the three I spoke about.

Same. Numba does wonders for me in most scenarios. Yesterday I've discovered pola-rs and looks like I will add it to the stack. It's API is similar to pandas. Have a look at the benchmarks of cuDF, spark, dask, pandas compared to it: Benchmarks

The syntax has similarities with dplyr in terms of the way you chain operations, and it’s around an order of magnitude faster than pandas and dplyr (there’s a nice benchmark here). It’s also more memory-efficient and can handle larger-than-memory datasets via streaming if needed.

If interested in benchmarks comparing different dataframe implementations, here is one:

https://h2oai.github.io/db-benchmark/

Bad examples. Both numpy and pandas are notoriously un-optimized packages, losing handily to pretty much all their competitors (R, Julia, kdb+, vaex, polars). See https://h2oai.github.io/db-benchmark/ for a partial comparison.

On include-- you might like FromFile.jl as an alternative.

For this and other oddities with Julia's include/import system (and especially as you're coming from Python), I'd recommend FromFile as a readable way to approach things.

You might like FromFile.jl.

However, if you prefer a Python-like experience, checkout FromFile.jl

I'm not using modules. I usually start with one file with a demo or similarly named function that is called if the file is called as an entry point (like if __name__ == '__main__', except Julia makes it even worse).

I tend to refactor code out of there to separate files, and then somehow import it. An ugly way is include, and I've tried Revise.jl with includet.

But I think the least ugly approach is the @from macro from here: https://github.com/Roger-luo/FromFile.jl Judging from some opinion in bug trackers, this is probably gonna get totally shunned by core devs and they'll keep on bikeshedding about the import stuff forever.

With this setup I have about 400 lines of code in three files. It compiles for 15 seconds. After every single change, and actually without any changes too.

I think performance wise this should be equivalent to using modules, but saving some pointless ceremony.