Analyzing multi-gigabyte JSON files locally

• xsv

  25 9,395 0.0 Rust

  A fast CSV command line toolkit written in Rust.

  If it could be tabular in nature, maybe convert to sqlite3 so you can make use of indexing, or CSV to make use of high-performance tools like xsv or zsv (the latter of which I'm an author).

  https://github.com/BurntSushi/xsv

  https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql...

• ClickHouse

  83 28,880 10.0 C++

  ClickHouse® is a free analytics DBMS for big data

  

  It is not necessary to write a code with Rust and simdjson. You can insert the dataset into ClickHouse as described here: https://github.com/ClickHouse/ClickHouse/issues/22482

  Running full scan for ClickHouse in body took 140 seconds on a single c5ad.24xlarge machine, the bottleneck is disk read (only 24 CPU cores are used).

• CodiumAI

  codium.ai

  sponsored

  TestGPT | Generating meaningful tests for busy devs. Get non-trivial tests (and trivial, too!) suggested right inside your IDE, so you can code smart, create more value, and stay confident when you push.

  

• octosql

  21 4,497 7.6 Go

  OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

• json-buffet

  2 0 10.0 C++

  

  And here's the code: https://github.com/multiversal-ventures/json-buffet

  The API isn't the best. I'd have preferred an iterator based solution as opposed to this callback based one. But we worked with what rapidjson gave us for the proof of concept.

• pysimdjson

  3 586 1.2 Python

  Python bindings for the simdjson project.

• reddit_mining

  4 11 10.0 HTML

  zstd decompression should almost always be very fast. It's faster to decompress than DEFLATE or LZ4 in all the benchmarks that I've seen.

  you might be interested in converting the pushshift data to parquet. Using octosql I'm able to query the submissions data (from the begining of reddit to Sept 2022) in about 10 min

  https://github.com/chapmanjacobd/reddit_mining#how-was-this-...

  Although if you're sending the data to postgres or BigQuery you can probably get better query performance via indexes or parallelism.

• json-streamer

  2 196 0.9 Python

  A fast streaming JSON parser for Python that generates SAX-like events using yajl

  Might be useful for some - https://github.com/kashifrazzaqui/json-streamer

• InfluxDB

  www.influxdata.com

  sponsored

  Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  

• json-toolkit

  1 57 0.0 Python

  "the best opensource converter I've found across the Internet" -- dene14

  > Also note that this approach generalizes to other text-based formats. If you have 10 gigabyte of CSV, you can use Miller for processing. For binary formats, you could use fq if you can find a workable record separator.

  You can also generalize it without learning a new minilanguage by using https://github.com/tyleradams/json-toolkit which converts csv/binary/whatever to/from json

• lnav

  45 5,660 9.2 C++

  Log file navigator

  LNAV (https://lnav.org) is ideally suited for this kind of thing, with an embedded sqlite engine and what amounts to a local laptop-scale mini-ETL toolkit w/ a nice CLI. I've been recommending it for the last 7 years since I discovered this awesome little underappreciated util.

• zsv

  17 121 8.1 C

  zsv+lib: world's fastest (simd) CSV parser, bare metal or wasm, with an extensible CLI for SQL querying, format conversion and more

  If it could be tabular in nature, maybe convert to sqlite3 so you can make use of indexing, or CSV to make use of high-performance tools like xsv or zsv (the latter of which I'm an author).

  https://github.com/BurntSushi/xsv

  https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql...

• semi_index

  1 53 10.0 C++

  Implementation of the JSON semi-index described in the paper "Semi-Indexing Semi-Structured Data in Tiny Space"

• jq-zsh-plugin

  1 190 10.0 Shell

  jq zsh plugin

  https://github.com/reegnz/jq-zsh-plugin

  I find that for big datasets choosing the right format is crucial. Using json-lines format + some shell filtering (eg. head, tail to limit the range, egrep or ripgrep for the more trivial filtering) to reduce the dataset to a couple of megabytes, then use that jq-repl of mine to iterate fast on the final jq expression.

  I found that the REPL form factor works really well when you don't exactly know what you're digging for.

Suggest a related project