Top 10 Rust Parquet Projects

roapi

24 3,069 6.9 Rust

Create full-fledged APIs for slowly moving datasets without writing a single line of code.

Project mention: Full-fledged APIs for slowly moving datasets without writing code | news.ycombinator.com | 2023-10-25

qsv

13 2,203 9.9 Rust

CSVs sliced, diced & analyzed.

Project mention: Qsv: Efficient CSV CLI Toolkit | news.ycombinator.com | 2023-12-22

Thanks for the detailed feedback @snidane!
As maintainer of qsv, here's my reply:
- Given qsv's rapid release cycle (173 releases over three years), the auto-update check is essential at the moment. Once we reach 1.0, I'll turn it off. For now, given your feedback, I've only made it check 10% of the time.
- Pivot is in the backlog and I'll be sure to add unpivot when I implement it. (https://github.com/jqnatividad/qsv/issues/799)
- I'll add a dedicated summing command with the group by (-by) and window by (-over) capability (https://github.com/jqnatividad/qsv/issues/1514). Do note that `stats` has basic sum as @ezequiel-garzon pointed out.
- With the `enum` command, qsv can achieve what you proposed with `laminate`. E.g. qsv enum --new-column newcol --constant newconstant mydata.csv --output laminated-data.csv
- With the cat rowskey command, qsv can already concatenate files with mismatched headers.
- other file formats. qsv supports parquet, csv, tsv, excel, ods, datapackage, sqlite and more (see https://github.com/jqnatividad/qsv/tree/master#file-formats). Fixed-format though is not supported yet and quite interesting, and have added it to the backlog (https://github.com/jqnatividad/qsv/issues/1515)
- as to "enable embedding outputs of commands", qsv is composable by design, so you can use standard stdin/stdout redirection/piping techniques to have it work with other CLI tools like jq, awk, etc.
Finally, just released v0.120.0 that already incorporates the less aggressive self-update check. https://github.com/jqnatividad/qsv/releases/tag/0.120.0

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
cryo

15 967 9.5 Rust

cryo is the easiest way to extract blockchain data to parquet, csv, json, or python dataframes

Project mention: cryo: NEW Data - star count:778.0 | /r/algoprojects | 2023-12-09

parquet2

6 347 3.2 Rust

Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow
pqrs

4 245 4.9 Rust

Command line tool for inspecting Parquet files
parquet-wasm

4 223 8.8 Rust

Rust-based WebAssembly bindings to read and write Apache Parquet data

Project mention: Goodbye, Node.js Buffer | news.ycombinator.com | 2023-10-24

nodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box.
As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)

odbc2parquet

5 204 9.3 Rust

A command line tool to query an ODBC data source and write the result into a parquet file.
WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
warc-parquet

4 99 7.0 Rust

🗄️ A simple CLI for converting WARC to Parquet.
dply-rs

1 37 8.0 Rust

A dataframe manipulation tool for parquet, csv, and json data.

Project mention: A dplyr interpreter powered by Polars | /r/rust | 2023-05-10

I have added documentation for all supported functions here.

bdt

1 6 5.8 Rust

Boring Data Tool (by andygrove)

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-12-22.