sqlite-utils VS ShellCheck

https://github.com/simonw/sqlite-utils

So, his code might not be a good place to find best patterns (for ex, I don't think they are fully typed), but his repos are very pragmatic, and his development process is super insightful (well documented PRs for personal repos!). Best part, he blogs about every non-trivial update, so you get all the context!

Sounds like your problem is with SQLAlchemy, not with SQLite.

My https://sqlite-utils.datasette.io library might be a better fit for you. It's a much thinner abstraction than SQLAlchemy.

There are a few things you can do here.

SQLite is great at JSON - so I often dump JSON structures in a TEXT column and query them using https://www.sqlite.org/json1.html

I also have plugins for running jq() functions directly in SQL queries - https://datasette.io/plugins/datasette-jq and https://github.com/simonw/sqlite-utils-jq

I've been trying to drive the cost of turning semi-structured data into structured SQL queries down as much as possible with https://sqlite-utils.datasette.io - see this tutorial for more: https://datasette.io/tutorials/clean-data

This is also an area that I'm starting to explore with LLMs. I love the idea that you could take a bunch of messy data, tell Datasette Cloud "I want this imported into a table with this schema"... and it does that.

I have a prototype of this working now, I hope to turn it into an open source plugin (and Datasette Cloud feature) pretty soon. It's using this trick: https://til.simonwillison.net/gpt3/openai-python-functions-d...

I've baked a ton of different SQLite tricks - including things like full-text indexing support and advanced alter table methods - into my sqlite-utils CLI tool and Python library: https://sqlite-utils.datasette.io

My Datasette project provides tools for exploring, analyzing and publishing SQLite databases, plus ways to expose them via a JSON API: https://datasette.io

I've also written a ton of stuff about SQLite on my two blogs:

- https://simonwillison.net/tags/sqlite/

- https://til.simonwillison.net/sqlite

This is really fun. I have an experimental branch of my sqlite-utils CLI tool (which has dozens of sub-commands) running with this now and it really did only take 4 lines of code - I'm treating Trogon as an optional dependency because people using my package as a Python library rather than a CLI tool may not want the extra installed components:

https://github.com/simonw/sqlite-utils/commit/ec12b780d5dcd6...

There's an animated GIF demo of the result here: https://github.com/simonw/sqlite-utils/issues/545#issuecomme...

My project https://datasette.io/ is ideal for this kind of thing. You can use https://sqlite-utils.datasette.io/ to load JSON data into a SQLite database, then publish it with Datasette.

I've been using this for about six months now and I absolutely love it.

Make never stuck for me - I couldn't quite get it to fit inside my head.

Just has the exact set of features I want.

Here's one example of one of my Justfiles: https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624... - documented here: https://sqlite-utils.datasette.io/en/stable/contributing.htm...

I also wrote about using Just with Django in this TIL: https://til.simonwillison.net/django/just-with-django

SQLite with the open source toolchain I've been building over the past five years:

https://datasette.io as the interface for running queries against (and visualizing) my data.

https://sqlite-utils.datasette.io/ as a set of tools for creating and modifying my databases (inserting JSON or CSV data, enabling full text search text)

https://dogsheep.github.io as a suite of tools for importing my personal data - see also this talk I gave about that project: https://simonwillison.net/2020/Nov/14/personal-data-warehous...

Here's an example: https://github.com/simonw/sqlite-utils/pull/468

> After identifying about 7 commits (with pretty basic/useless messages, and no PR link!), I then had to find the corresponding PRs based on timestamps, and search the PR history for PRs merged around those timestamps.

Not sure if this would save any time, but it is possible to search PRs by commit. For example, say git blame led me to this commit: https://github.com/simonw/sqlite-utils/commit/129141572f249e...

I could have found PR #373 via this search: https://github.com/simonw/sqlite-utils/pulls?q=bb16f52681b6d...

> I thus treat PRs as ephemeral

I think I see what you're saying but as others have pointed out, sometimes you want to add screenshots etc to the context, and you can't capture this kind of info in commit messages. So then you have two choices: issues or PRs.

> Then any review comments are preferably not addressed directly in the PR

I would think that sometimes you really do want to have a back and forth conversation in the PR, rather than just a "make this change" -> "ok done" type of feedback loop.

I view the PR as an decent place for all of this because it's basically a commit of commits, capturing the related changes/conversation/context all in a single place at the point of merge.

These projects use Caddy as my local development server, Dart Sass for converting my Sass files to CSS, elm, elm-format, elm-optimize-level-2, elm-review, elm-test (only in Calculator), ShellCheck to find bugs in my shell scripts, and Terser to mangle and compress JavaScript code.

ncurse, dialog, zenity[2]. i/o buffering may be an issue [3a,3b]

Assuming using same account, use history command to show past commands[0a, 0b]

'load random example' on shellcheck using own custom examples from history command.[1]

--------

[3a] : http://www.gnu.org/software/coreutils/manual/html_node/stdbu...

[3b] : http://unix.stackexchange.com/questions/25372/how-to-turn-of...

[2] : http//funprojects.blog/2021/01/25/zenity-command-line-dialogs/

[1] : http://www.shellcheck.net/

[0a] : http://www.tecmint.com/history-command-examples/

[0b] : http://www.tecmint.com/remember-linux-commands/

web based documentation: https://www.tecmint.com/linux-commands-cheat-sheet/

commands grouped by typical usage patterns : https://www.tecmint.com/essential-linux-commands/

... #************************** Terraform ************************************* ARG TERRAFORM_VERSION=1.7.3 RUN set -ex \ && curl -O https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin/ RUN set -ex \ && mkdir -p $HOME/.terraform.d/plugin-cache && echo 'plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"' > ~/.terraformrc #************************* Terragrunt ************************************* ARG TERRAGRUNT_VERSION=0.55.1 RUN set -ex \ && wget https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 -q \ && mv terragrunt_linux_amd64 /usr/local/bin/terragrunt \ && chmod +x /usr/local/bin/terragrunt #*********************** Terramate **************************************** ARG TERRAMATE_VERSION=0.4.5 RUN set -ex \ && wget https://github.com/mineiros-io/terramate/releases/download/v${TERRAMATE_VERSION}/terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && tar -xzf terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && mv terramate /usr/local/bin/terramate \ && chmod +x /usr/local/bin/terramate #*********************** tfsec ******************************************** ARG TFSEC_VERSION=1.28.5 RUN set -ex \ && wget https://github.com/aquasecurity/tfsec/releases/download/v${TFSEC_VERSION}/tfsec-linux-amd64 \ && mv tfsec-linux-amd64 /usr/local/bin/tfsec \ && chmod +x /usr/local/bin/tfsec \ && terragrunt --version #**********************Terraform docs ************************************ ARG TERRRAFORM_DOCS_VERSION=0.17.0 RUN set -ex \ && curl -sSLo ./terraform-docs.tar.gz https://terraform-docs.io/dl/v${TERRRAFORM_DOCS_VERSION}/terraform-docs-v${TERRRAFORM_DOCS_VERSION}-$(uname)-amd64.tar.gz \ && tar -xzf terraform-docs.tar.gz \ && chmod +x terraform-docs \ && mv terraform-docs /usr/local/bin/terraform-docs #********************* ShellCheck ***************************************** ARG SHELLCHECK_VERSION="stable" RUN set -ex \ && wget -qO- "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION?}/shellcheck-${SHELLCHECK_VERSION?}.linux.x86_64.tar.xz" | tar -xJv \ && cp "shellcheck-${SHELLCHECK_VERSION}/shellcheck" /usr/bin/ \ && shellcheck --version ...

ShellCheck: https://github.com/koalaman/shellcheck

If I want to write better shell scripts I usually run shellcheck and adjust accordingly or if I need facilities not provided by the shell i switch to a full fledged programming language. Ans oh yes, `sh` is present almost on every BSD and Linux box for free so I consider it an important thing to at least be comfortable with.

shellcheck: https://www.shellcheck.net/

When I run nix-shell at the root of the project it puts me in a Nix shell that contains, among other programs, caddy and shellcheck. Notice that in the shellHook I add the project's shell scripts to the PATH. So once I'm in the Nix shell I can, among other things:

I have also seen some projects on github like ShellCheck which first make a library, expose all the modules and then simple add that do build-depends of the final executable. Is this the recommended approach than having just one executable and adding all the modules to other-modules:?

The error checks can be pretty arcane:

   https://github.com/koalaman/shellcheck/wiki/Checks

Similar to for instance shellcheck to check the syntax of shell scripts, is there an equivalent for the set of roff commands typically used in a (Linux) man page? I'm aware that e.g. pandoc permits the conversion of an other format (e.g., org) to both roff man and roff ms.