reddit_mining
lnav
| reddit_mining | lnav | |
|---|---|---|
| 4 | 77 | |
| 11 | 6,727 | |
| - | - | |
| 2.6 | 9.6 | |
| 10 months ago | 4 days ago | |
| HTML | C++ | |
| Creative Commons Zero v1.0 Universal | BSD 2-clause "Simplified" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
reddit_mining
-
Analyzing multi-gigabyte JSON files locally
zstd decompression should almost always be very fast. It's faster to decompress than DEFLATE or LZ4 in all the benchmarks that I've seen.
you might be interested in converting the pushshift data to parquet. Using octosql I'm able to query the submissions data (from the begining of reddit to Sept 2022) in about 10 min
https://github.com/chapmanjacobd/reddit_mining#how-was-this-...
Although if you're sending the data to postgres or BigQuery you can probably get better query performance via indexes or parallelism.
- reddit_mining - List of all Subreddits
- Show HN: List of All Subreddits
- Top 50k Subreddits
lnav
- Lnav: A log file viewer for the terminal
-
Angle-grinder: Slice and dice logs on the command line
See https://lnav.org for a powerful mini-ETL CLI power tool; it embeds SQLite, supports ~every format, has great UX and easily handles a few million rows at a time.
- FLaNK Stack 26 February 2024
- LNAV – The Logfile Navigator
-
Toolong: Terminal application to view, tail, merge, and search log files
The code base seems like a good reference as a small Python project.
My fav option in this class of apps: https://lnav.org/ It lets you use journalctl with pipes as requested here: https://github.com/Textualize/toolong/issues/4
-
Logdy.dev – web based logs viewer UI for local development environment
For local development, I cannot recommend lnav[1] enough. Discovering this tool was a game changer in my day to day life. Adding comments, filtering in/out, prettify and analyse distribution is hard to live without now.
I don't think a browser tool would fit in my workflow. I need to pipe the output to the tool.
[1] https://lnav.org/
- Textanalysistool.net
- Ask HN: What apps have you created for your own use?
-
Ask HN: How does `lnav` run its playground which you can just SSH into?
It looks like they run an SSH server inside a Docker container defined by this Dockerfile [1]. This uses the ForceCommand directive in the sshd_config file to ensure that a specific command is run when a user connects (rather than the user connecting directly to a shell).
Depending on whether the user connects as the `playground` or `tutorial1` user they interact with a bash script that is either [2] or [3].
[1]: https://github.com/tstack/lnav/blob/master/demo/Dockerfile
[2]: https://github.com/tstack/lnav/blob/master/docs/tutorials/pl...
[3]: https://github.com/tstack/lnav/blob/master/docs/tutorials/tu...
What are some alternatives?
json-streamer - A fast streaming JSON parser for Python that generates SAX-like events using yajl
lightproxy - 💎 Cross platform Web debugging proxy
json-buffet
dive - A tool for exploring each layer in a docker image
semi_index - Implementation of the JSON semi-index described in the paper "Semi-Indexing Semi-Structured Data in Tiny Space"
glow - Render markdown on the CLI, with pizzazz! 💅🏻
jq-zsh-plugin - jq zsh plugin
GoAccess - GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
octosql - OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
conio-for-linux - Conio.h for linux
zsv - zsv+lib: tabular data swiss-army knife CLI + world's fastest (simd) CSV parser
nnn - n³ The unorthodox terminal file manager