json-streamer
jq-zsh-plugin
json-streamer | jq-zsh-plugin | |
---|---|---|
2 | 4 | |
215 | 298 | |
- | - | |
2.4 | 6.0 | |
about 1 year ago | 21 days ago | |
Python | Shell | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
json-streamer
- Processing large JSON datasets by streaming
-
Analyzing multi-gigabyte JSON files locally
Might be useful for some - https://github.com/kashifrazzaqui/json-streamer
jq-zsh-plugin
- Interactive Examples for Learning Jq
-
Analyzing multi-gigabyte JSON files locally
https://github.com/reegnz/jq-zsh-plugin
I find that for big datasets choosing the right format is crucial. Using json-lines format + some shell filtering (eg. head, tail to limit the range, egrep or ripgrep for the more trivial filtering) to reduce the dataset to a couple of megabytes, then use that jq-repl of mine to iterate fast on the final jq expression.
I found that the REPL form factor works really well when you don't exactly know what you're digging for.
What are some alternatives?
ijson
semi_index - Implementation of the JSON semi-index described in the paper "Semi-Indexing Semi-Structured Data in Tiny Space"
python-slugify - Returns unicode slugs
z-a-readurl - 🌀 An annex delivers the capability to automatically download the newest version of a file to which URL is hosted on a webpage
awesome-slugify - Python flexible slugify function
json-buffet
python-nameparser - A simple Python module for parsing human names into their individual components
lnav - Log file navigator
Lark - Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.
reddit_mining
pydantic - Data validation using Python type hints
ClickHouse - ClickHouse® is a free analytics DBMS for big data