Command-line data analytics made easy

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • arrow-datafusion

    Apache Arrow DataFusion SQL Query Engine

    SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions!

    As far as similar tools go, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2].

    DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats.

    dsq is based on SQLite which means it has to load data into SQLite first, but then gives you the whole breath of SQLite, it also supports many data formats, but is slower at the same time.

    OctoSQL is faster, extensible through plugins, and supports incremental query execution, so you can i.e. calculate a running group by + count while tailing a log file. It also supports normal databases, not just file formats, so you can i.e. join with a Postgres table.

    [0]: https://github.com/apache/arrow-datafusion

    [1]: https://github.com/multiprocessio/dsq

    [2]: https://github.com/cube2222/octosql

    Disclaimer: Author of OctoSQL

  • roapi

    Create full-fledged APIs for slowly moving datasets without writing a single line of code.

    It could be the NDJSON parser (DF source: [0]) or could be a variety of other factors. Looking at the ROAPI release archive [1], it doesn't ship with the definitive `columnq` binary from your comment, so it could also have something to do with compilation-time flags.

    FWIW, we use the Parquet format with DataFusion and get very good speeds similar to DuckDB [2], e.g. 1.5s to run a more complex aggregation query `SELECT date_trunc('month', tpep_pickup_datetime) AS month, COUNT(*) AS total_trips, SUM(total_amount) FROM tripdata GROUP BY 1 ORDER BY 1 ASC)` on a 55M row subset of NY Taxi trip data.

    [0]: https://github.com/apache/arrow-datafusion/blob/master/dataf...

    [1]: https://github.com/roapi/roapi/releases/tag/roapi-v0.8.0

    [2]: https://observablehq.com/@seafowl/benchmarks

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • dsq

    Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

    SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions!

    As far as similar tools go, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2].

    DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats.

    dsq is based on SQLite which means it has to load data into SQLite first, but then gives you the whole breath of SQLite, it also supports many data formats, but is slower at the same time.

    OctoSQL is faster, extensible through plugins, and supports incremental query execution, so you can i.e. calculate a running group by + count while tailing a log file. It also supports normal databases, not just file formats, so you can i.e. join with a Postgres table.

    [0]: https://github.com/apache/arrow-datafusion

    [1]: https://github.com/multiprocessio/dsq

    [2]: https://github.com/cube2222/octosql

    Disclaimer: Author of OctoSQL

  • spyql

    Query data on the command line with SQL-like SELECTs powered by Python expressions

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts