-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
stern
⎈ Multi pod and container log tailing for Kubernetes -- Friendly fork of https://github.com/wercker/stern (by stern)
My biggest quality of life improvement for understanding logs has been lnav (https://lnav.org/) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.
I found that most of the "analysis" I perform ends up essentially just performing a search, but I never liked how `tail .. | grep ...` required you to run a new command to change the search criteria. I ended up prototyping my own tool for this [0], which is currently in alpha.
[0]: https://github.com/ReagentX/Logria
Thanks for linking to this!
Until now I thought logview.el[0] is the bee's knees, but now I can feel feature envy set in. There are some seriously powerful ideas listed on the lnav page, and it's also the first time I saw SQLite virtual tables used in the wild.
--
[0] - https://github.com/doublep/logview
Loosely related: a few years ago I wanted a simpler alternative to some of the more feature-full log viewers out there so I threw together a tiny app that might be useful to some folks in here. All it does is consistently colors the first field in a line from stdin so you can quickly see which log lines have the same first field. I used it in combination with the parallel command to prefix log lines by replica name when tailing logs across machines: https://github.com/jasisk/color-prefix-pipe
I wrote https://github.com/ljw1004/seaoflogs - an interactive filtering tool, for similar ends to what's described here. I wrote it because my team was struggling to analyze LSP logs (that's the protocol used by VSCode to communicate with language servers). But I made it general-purpose able to analyze more log formats too - for instance, we want to correlate LSP logs with server logs and other traffic logs.
(1) I wanted something where colleagues could easily share links in workplace chat with each other, so we could cooperatively investigate bugs.
(2) For LSP we're often concerned with responsiveness, and I thought the best way to indicate times when viewing a log is with whitespace gaps between log messages in proportion to their time gap.
(3) For LSP we have lots of interleaved activity going on, and I wanted to have visual "threads" connecting related logs.
(4) As the post and lnav say, interactivity is everything. I tried to take it a step further with (1) javascript, (2) playground-style updates as you type, (3) autocomplete which "learns" what fields are available from structured logs.
My tool runs all in the browser. (I spent effort figuring out how people can distribute it safely and use it for their own confidential logs too). It's fast enough up to about 10k lines of logs.
If you are using Kubernetes, I highly recommend using https://github.com/stern/stern