DataFrames.jl VS Tables.jl

I don't think we've seen the final state of it though. v1.9 really gives people the proper tools for solving latency problems. Before, invalidations hurt a little, but it was also kind of a wash because LLVM code didn't precompile, so you could spend time setting up a Snoopprecompile and fix some invalidations and end up LLVM bound saving 1 second out of 20. But with package images it's almost always better to fix precompilation. The only thing hampering time much now is the `using` time went up, but as mentioned in the Reddit post there's a lot of ideas for what to do there. The other thing is package extensions, which cut down the amount of code to load. There's tons of PRs floating around the ecosystem turning things into extensions, and thus cutting down the overall code that is actually ran and loaded.

This means that in a few months, people will start to see some major tangible benefits from following the compilation improvement practices laid out here and https://sciml.ai/news/2022/09/21/compile_time/. I think then you'll have a lot more people start to take all of these new tools seriously and it will be standard to incorporate them into packages. Right now they are still kind of niche things for packages with known TTFX problems, but I think come v1.9 you'll see every major package use all of these methods.

> Different packages are affected differently

I think this is one of the pieces that's effected by this. I don't think "Julia has become optimised for running Plots.jl" is quite correct. Julia's compilation and runtime is much more optimized for well-inferred code, which Plots.jl is not. However, the compiler developers have been using Plots.jl as a test case for all of these new tools, and therefore its SnoopPrecompile and invalidations have gotten some dramatic improvements because those required ecosystem changes I mentioned are being done by the compiler team for this specific package. Plots.jl had things like precompilation snooping way back before there was even a package for it, the earliest I know of was around v1.0. Meanwhile, DataFrames.jl only setup its precompilation snooping 7 months ago (https://github.com/JuliaData/DataFrames.jl/pull/3182), which I would presume was just in time for the v1.8 mark on your plot and is one of the big reasons for having a sudden drop (which continues into v1.9 because of package images).

What I mean to say then is that, I think all packages will get the improvements we've seen from Plots.jl, but package authors will need to update their packages in order for that to happen. Some packages have already done this, many have not.

IJulia also supports viewing and manipulating tables. To create a table, first install the DataFrames.jl package by running the following command in a new cell:

It were just a few percents of all possible manipulations that you can do with data using DataFrames.jl library. Read more about it in the documentation.

DataFrames.jl

DataFrames.jl and XLSX.jl for JSON, CSV, and XLSX files

Julia also has the CSV.jl library for reading/writing csv files, the DataFrames.jl library for manipulating data like pandas, and Images.jl for image processing/analysis. However, since Julia is so much newer than Python, the Julia libraries are almost never as feature rich as their Python counterparts.

Looks like it's single threaded according to this open issue: https://github.com/JuliaData/DataFrames.jl/issues/2626

Good questions!

> How exactly does Julia fit into your software architecture?

In a variety of ways:

- We have a bunch of external/internal Julia packages; Julia's package manager is really great at facilitating the development of "tooling ecosystems" comprised of lightweight libraries that compose well together. For example, we use Legolas.jl [1] in conjunction with a well-curated Arrow-in-S3 lake to help teams define lightweight, self-serviceable schemas for Arrow tables in a manner that integrates well with the wider Tables.jl ecosystem [2], interactive analysis workflows, and our own ETL/ELT-ish patterns.

- Julia powers some interesting services within Beacon's Platform. For example, one of our Julia services provides dynamic streaming DSP (multiplexing, filtering, statistics) for biosignal data, atop which we build other applications/pipelines for both product development and internal analysis work.

- We use Julia for exploratory distributed computing on K8s [3], which is awesome because Julia has a lot of potential in the distributed computing landscape (IMO [4]).

> Is your product a cloud offering and/or does it have a client side application?

We work with our clients to do neurobiomarker discovery, clinical trial design, deploy our analysis pipelines into clinical trials, and a few other interesting things :) One of the critical differentiators of Beacon is that we can precisely target and harness key EEG features to a degree that isn't possible without the kind of algorithms/tools we've developed.

> what do you even mean by data architecture for science-first teams

I want to do a blog post on this at some point, but a core value for us - across all of our processes, tooling, and data interactions - is self-serviceability and composability. IMO, the two are inextricably linked. Our goal is to empower each Beaconeer to perform analyses in an afternoon atop terabytes of data that would take them months in a lab atop gigabytes of data.

To achieve this, we treat large-scale data curation/manipulation as an activity that we're all empowered to participate in and contribute to, as opposed to an environment where separate data engineering teams have to administrate siloed systems. Tools like K8s/Julia/Arrow are key enablers here, by surfacing capabilities to domain experts that let them to iterate fast without needing to "throw problems over the wall" to other teams/systems.

It's not a perfect match, and it's a bit abstract, but I remember reading this post about "data meshes" [5] a while back and thinking "Hey, that's similar to what we're chasing after!"

[1] https://github.com/beacon-biosignals/Legolas.jl

[2] https://github.com/JuliaData/Tables.jl

[3] https://github.com/beacon-biosignals/K8sClusterManagers.jl

[4] https://news.ycombinator.com/item?id=24842084

[5] https://martinfowler.com/articles/data-mesh-principles.html

As you progress you'll likely learn to be a bit more relaxed about types - there's a Table Interface that JuliaDB implements along with many other data sources.But this should get you going.