miller VS scc

jq and miller[1] are essential parts of my toolbelt, right up there with awk and vim.

[1]: https://github.com/johnkerl/miller

> This works really well if your problem can be solved in one or two liners.

My personal comfort threshold is around the 100-line mark. It's even possible to write maintainable shell scripts up to 500 lines, but it mostly depends on the problem you're trying to solve, and the discipline of the programmer to follow best practices (use sane defaults, ShellCheck, etc.).

> It go bad very quickly when, say, you have two CSV files and want to join them the sql-way.

In that case we're talking about structured data, and, yeah, Perl or Python would be easier to work with. That said, depending on the complexity of the CSV, you can still go a long way with plain Bash with IFS/read(1) or tr(1) to split CSV columns. This wouldn't be very robust, but there are tools that handle CSV specifically[1], which can be composed in a shell script just fine.

So it's always a balancing act of being productive quickly with a shell script, or reaching out for a programming language once the tools aren't a good fit, or maintenance becomes an issue.

[1]: https://miller.readthedocs.io/

where mlr is from https://github.com/johnkerl/miller

if this class of problems (i.e., csv/tsv data) is your main target you may find miller (https://github.com/johnkerl/miller) much more useful in the long run

That said, you may be interested in Miller (https://github.com/johnkerl/miller) which provides similar capabilities for CSV, JSON, and XML files. It doesn't use a SQL grammar, but that's just the proverbial lipstick on the thing. I'm not the author, but I have used it and I see some parallels in use cases at the very least.

worth giving Miller a shot

CSV utilities (still haven't pick a favorite one...): https://github.com/harelba/q https://github.com/BurntSushi/xsv https://github.com/wireservice/csvkit https://github.com/johnkerl/miller

View on GitHub

Going to say my own https://github.com/boyter/scc/ which I have used to turn down projects of "Oh we just need to do X"

It allows me to evaluate the code-base quickly and see where potential issues are, and find hidden complexity in the code. I have said no a lot due to it. The only reason it exists was because I got caught out from another project, which wasted months of my time.

Otherwise IntelliJ and the JetBrains IDE's in general.

Very cool to see this here, Ben! It was fun beating the ins and outs of your work on this in the TZ discord.

Also, off-topic but as you know, I recently tried out your scc tool and am eagerly awaiting its support for Elixir templates (.eex, .heex)!

https://github.com/boyter/scc

I believe tokei is the best rust option as of now, but despite my burning passion for rust I've switched to using scc instead as I find it faster and more convenient. Not really an option for you if you're trying to bake line counting into the binary, obviously.

I've seen some real world example where Go was as fast or faster than Rust for CPU / io intensive task.

Go is a fast language even with a GC.

https://github.com/boyter/scc/#performance

I don't trust your qualitative "LARGE" for the project. I would recommend you pass your project through something like a software metrics tool https://github.com/boyter/scc to better measure what you're up against in terms of Flutter/Dart AND Rust code base.

BTW, for technically inclined, here is scc output for an exported inventory tracker app (you can see actual app build @ https://www.bappaz.com/video or step by step process to build the app @ https://www.bappaz.com/learn )