lnav VS octosql

There’s a tool called lnav that will parse logfiles into a temporary SQLite database and allows to analyse them using SQL features:

https://lnav.org/

It creates a patch file since the original file might've been modified.

> - There are lots of different filtering capabilities, but there is no unified treatment of them. For example, `:hide-lines-before` and `:filter-out` are at their core the same type of operation: filtering. I should be able to pull up a list of all filters that are currently active and easily add new ones and toggle or delete existing ones.

Adding the time filters to the "Filters" panel sounds like a reasonable request. I've added https://github.com/tstack/lnav/issues/1275 to track.

> - I would expect to be able to create a new view of the data using SQL `SELECT`. A select statement is fundamentally about filtering out some rows (log lines), which feels like a filter, and selecting some particular columns (log fields) and hiding others. The latter point seems like it could be something that should be handled when https://github.com/tstack/lnav/issues/1274 is resolved.

There is the `:filter-expr` command (https://docs.lnav.org/en/v0.12.2/commands.html#filter-expr-e...), have you tried that?

As others have kinda alluded to, it could be useful for testing TUI applications. I develop a logfile viewer for the terminal (https://lnav.org) and have a similar application[1] for testing, but it's a bit flaky. It produces/checks snapshots like [2]. I think the problems I run into are more around different versions of ncurses producing slightly different outputs.

[1] - https://github.com/tstack/lnav/blob/master/test/scripty.cc

The Logfile Navigator (https://lnav.org) is a log file viewer/merger/tailer for the terminal. It has some advanced UX features, like showing previews of operations and displaying context sensitive help. For example, the preview for filtering out logs by regex is to highlight the lines that will be hidden in red. This can make crafting the right regex a bit easier since the preview updates as you type. lnav also has some simple bar charting abilities, so you can visualize the results of SQL queries made against the log messages.

See https://lnav.org for a powerful mini-ETL CLI power tool; it embeds SQLite, supports ~every format, has great UX and easily handles a few million rows at a time.

The code base seems like a good reference as a small Python project.

My fav option in this class of apps: https://lnav.org/ It lets you use journalctl with pipes as requested here: https://github.com/Textualize/toolong/issues/4

Never got it to anything close to a finished state, instead moving on to doing the same prototype in llvm and then cranelift.

That said, here's some of the wazero-based code on a branch - https://github.com/cube2222/octosql/tree/wasm-experiment/was...

It really is just a very very basic prototype.

This is really cool!

With their Postgres scanner[0] you can now easily query multiple datasources using SQL and join between them (i.e. Postgres table with JSON file). Something I strived to build with OctoSQL[1] before.

It's amazing to see how quickly DuckDB is adding new features.

Not a huge fan of C++, which is right now used for authoring extensions, it'd be really cool if somebody implemented a Rust extension SDK, or even something like Steampipe[2] does for Postgres FDWs which would provide a shim for quickly implementing non-performance-sensitive extensions for various things.

Godspeed!

[0]: https://duckdb.org/2022/09/30/postgres-scanner.html

[1]: https://github.com/cube2222/octosql

[2]: https://steampipe.io

Congrats on the Show HN!

It's great to see more tools in this area (querying data from various sources in-place) and the Lambda use case is a really cool idea!

I've recently done a bunch of benchmarking, including ClickHouse Local and the usage was straightforward, with everything working as it's supposed to.

Just to comment on the performance area though, one area I think ClickHouse could still possibly improve on - vs OctoSQL[0] at least - is that it seems like the JSON datasource is slower, especially if only a small part of the JSON objects is used. If only a single field of many is used, OctoSQL lazily parses only that field, and skips the others, which yields non-trivial performance gains on big JSON files with small queries.

Basically, for a query like `SELECT COUNT(*), AVG(overall) FROM books.json` with the Amazon Review Dataset, OctoSQL is twice as fast (3s vs 6s). That's a minor thing though (OctoSQL will slow down for more complicated queries, while for ClickHouse decoding the input is and remains the bottleneck).

[0]: https://github.com/cube2222/octosql

To add somewhat of a counterpoint to the other response, I've tried the Steampipe CSV plugin and got 50x slower performance vs OctoSQL[0], which is itself 5x slower than something like DataFusion[1]. The CSV plugin doesn't contact any external API's so it should be a good benchmark of the plugin architecture, though it might just not be optimized yet.

That said, I don't imagine this ever being a bottleneck for the main use case of Steampipe - in that case I think the APIs themselves will always be the limiting part. But it does - potentially - speak to what you can expect if you'd like to extend your usage of Steampipe to more than just DevOps data.

[0]: https://github.com/cube2222/octosql

[1]: https://github.com/apache/arrow-datafusion

Disclaimer: author of OctoSQL

Actually, folks just use gRPC or Yaegi in Go.

See Terraform[0], Traefik[1], or OctoSQL[2].

Although I agree plugins would be welcome, especially for performance reasons, though also to be able to compile and load go code into a running go process (JIT-ish).

[0]: https://github.com/hashicorp/terraform

[1]: https://github.com/traefik/traefik

[2]: https://github.com/cube2222/octosql

Disclaimer: author of OctoSQL

Through more magic, you COULD of course use stuff like Spark, or easier with programs like TextQL, sq, OctoSQL.

The logo was created for OctoSQL and in the article you can find a lot of sample phrase-image combinations, as it describes the whole path (generation, variation, editing) I went down. Let me know what you think!