Superintendent.app: A desktop app for working with large CSV files using SQL. v2.0 now can load 1GB file in 20s and support regex/date parsing!

This page summarizes the projects mentioned and recommended in the original post on /r/programming
JSON Python Logs json-parser SQL

InfluxDB
Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • q

    q - Run SQL directly on delimited files and multi-file sqlite databases (by harelba)

    Superintendent.app pushes all computation to C (e.g. importing/exporting/querying). That's how we can load 1GB file within 20 seconds. Running `SELECT COUNT(*) FROM table` after importing takes 1 second to run. Q has its own benchmark, but it's hard to compare (see: https://github.com/harelba/q/blob/master/test/BENCHMARK.md)

  • klogg

    Really fast log explorer based on glogg project

    For example, the file star2002-full.csv from https://sdm.lbl.gov/fastbit/data/samples.html is a 1.99GB file and it takes less than 10 seconds to load in Notepad+. It's almost instant in https://github.com/variar/klogg.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • simdjson

    Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts