arrow-julia
RecursiveArrayTools.jl
Our great sponsors
| arrow-julia | RecursiveArrayTools.jl | |
|---|---|---|
| 4 | 3 | |
| 277 | 198 | |
| 1.8% | 1.5% | |
| 6.2 | 9.4 | |
| 16 days ago | 19 days ago | |
| Julia | Julia | |
| GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
arrow-julia
-
Julia 1.8 has been released
For some examples of people porting existing C++ Fortran libraries to julia, you should check out https://github.com/JuliaLinearAlgebra/Octavian.jl, https://github.com/dgleich/GenericArpack.jl, https://github.com/apache/arrow-julia (just off the top of my head). These are all ports of C++ or Fortran libraries that match (or exceed) performance of the original, and in the case of Arrow.jl is faster, more general, and 10x less code.
- How to adapt Arrow.Table columns (naturally per record batch basis) into CuArrays for GPU processing?
-
Reading HDF5 Files
I guess current preferred format not feather, but arrow: https://github.com/JuliaData/Arrow.jl
-
Apache Arrow 3.0.0 Release
Excited to see this release's official inclusion of the pure Julia Arrow implementation [1]!
It's so cool to be able mmap Arrow memory and natively manipulate it from within Julia with virtually no performance overhead. Since the Julia compiler can specialize on the layout of Arrow-backed types at runtime (just as it can with any other type), the notion of needing to build/work with a separate "compiler for fast UDFs" is rendered obsolete.
It feels pretty magical when two tools like this compose so well without either being designed with the other in mind - a testament to the thoughtful design of both :) mad props to Jacob Quinn for spearheading the effort to revive/restart Arrow.jl and get the package into this release.
[1] https://github.com/JuliaData/Arrow.jl
RecursiveArrayTools.jl
-
Julia's latency: Past, present and future
You're not really supposed to be using StaticArraysCore anymore, but here's a somewhat older PR that shows the siginificance of moving StaticArray functionality on a smaller library, moving it from 6228ms to 292ms load time (https://github.com/SciML/RecursiveArrayTools.jl/pull/217).
-
Julia 1.8 has been released
> > This gives the package authors a tool to basically "profile" the loading time of their package, which will help them optimize the loading time. So there _will_ be downstream improvement to package loading for us users too.
It lead to https://github.com/SciML/RecursiveArrayTools.jl/pull/217 . 6228.5 ms to 292.7 ms isn't too shabby.
-
“Why I still recommend Julia”
The load times on some core packages were reduced by an order of magnitude this month. For example, RecursiveArrayTools went from 6228.5 ms to 292.7 ms. This was due to the new `@time_imports` in the Julia v1.8-beta helping to isolate load time issues. See https://github.com/SciML/RecursiveArrayTools.jl/pull/217 . This of course doesn't mean load times have been solved everywhere, but we now have the tooling to identify the root causes and it's actively being worked on from multiple directions.
What are some alternatives?
perspective - A data visualization and analytics component, especially well-suited for large and/or streaming datasets.
SciMLStyle - A style guide for stylish Julia developers
Apache Arrow - Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
Lux.jl - Explicitly Parameterized Neural Networks in Julia
arquero - Query processing and transformation of array-backed data tables.
ProtoStructs.jl - Easy prototyping of structs
ClickHouse - ClickHouse® is a free analytics DBMS for big data
ObjectOriented.jl - Conventional object-oriented programming in Julia without breaking Julia's core design ideas
TableIO.jl - A glue package for reading and writing tabular data. It aims to provide a uniform api for reading and writing tabular data from and to multiple sources.
SciMLSensitivity.jl - A component of the DiffEq ecosystem for enabling sensitivity analysis for scientific machine learning (SciML). Optimize-then-discretize, discretize-then-optimize, adjoint methods, and more for ODEs, SDEs, DDEs, DAEs, etc.
vega-loader-arrow - Data loader for the Apache Arrow format.
ITensors.jl - A Julia library for efficient tensor computations and tensor network calculations