Our great sponsors
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
bash-toolkit
Could be my ever-growing, ever-improving, Swiss Army Toolkit of functions-as-cmd-line-tools and useful-to-me patterns.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
r53
Python 3 alternative command line interface for AWS Route 53; enables simple record management and dynamic DNS
-
ripgrep
ripgrep recursively searches directories for a regex pattern while respecting your gitignore
-
plaintextaccounting
The plaintextaccounting.org website, a portal to Ledger, hledger, beancount and co. Also the PTA wiki.
-
misc-updater
Check if 'Manually-Installed and Source-Compiled' (MISC) packages have new releases or updates announced on their respective webpages.
-
display-brightness-scripts
Linux Shell scripts for increasing and decreasing display brightness, intended for use with solaar
-
-
shell-safe-rm
😎 Safe-rm: A drop-in and much safer replacement of bash rm with nearly full functionalities and options of the rm command! Safe-rm will act exactly the same as the original rm command.
-
shell-snippets
Some Bash shell snippets, with functions and utilities that I use frequently in my scripts.
-
-
-
-
-
-
fresh-mac
collection of setup scripts & default app installer for Mac OS, consistently up to date... ish.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
A few weeks ago, I asked if I could see your cheatsheets (https://news.ycombinator.com/item?id=31928736) and I was really impressed by all the high quality responses!
Today I'm asking about your scripts.
Almost every engineer I know has a collection of scripts/utilities for automating ${something}. I'm willing to bet that HN users have some of the most interesting scripts on the internet.
So that said, could I please see your scripts?
I'll go first: https://github.com/fastily/autobots
Going to use this opportunity to spam ShellCheck, because it has historically saved me dozens of hours catching many silent Bash scripting errors and just making my scripts more robust/warning me of obscure edge cases:
https://www.shellcheck.net/
}
It uses various linux utilities including fzf and batcat(https://github.com/sharkdp/bat) to open a terminal with all the places where my query comes up.
So i will do `search_notes postgres authentication`. I can select a line and it will open the file in less. Works like a charm!
> One I was particularly proud of/disgusted by
I can relate! I think it just reflects the nature of the problem space. The script is gnarly because the thing one is trying to do is gnarly. Utility is the driving force, as far as I'm concerned.
The following aren't as gnarly as yours, but served their purpose nicely in that little project's context. I like to put/accumulate project-related automations in a `./bin` in my projects.
https://gitlab.com/nilenso/cats/-/tree/master/bin
Here's my dotfiles repository [1], which is used to sync my little scripts and config files between my different systems (Mac/Linux). I first heard about it here [2].
[1] https://github.com/benwinding/dotfiles
[2] https://zachholman.com/2010/08/dotfiles-are-meant-to-be-fork...
Here's my dotfiles repository [1], which is used to sync my little scripts and config files between my different systems (Mac/Linux). I first heard about it here [2].
[1] https://github.com/benwinding/dotfiles
[2] https://zachholman.com/2010/08/dotfiles-are-meant-to-be-fork...
My dotfiles: https://github.com/BurntSushi/dotfiles
Here are some selected scripts folks might find interesting.
Here's my backup script that I use to encrypt my data at rest before shipping it off to s3. Runs every night and is idempotent. I use s3 lifecycle rules to keep data around for 6 months after it's deleted. That way, if my script goofs, I can recover: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...
I have so many machines running Archlinux that I wrote my own little helper for installing Arch that configures the machine in the way I expect: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...
A tiny little script to recover the git commit message you spent 10 minutes writing, but "lost" because something caused the actual commit to fail (like a gpg error): https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...
A script that produces a GitHub permalink from just a file path and some optional file numbers. Pass --clip to put it on your clipboard: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae... --- I use it with this vimscript function to quickly generate permalinks from my editor: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...
A wrapper around 'gh' (previously: 'hub') that lets you run 'hub-rollup pr-number' and it will automatically rebase that PR into your current branch. This is useful for creating one big "rollup" branch of a bunch of PRs. It is idempotent. https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...
Scale a video without having to memorize ffmpeg's crazy CLI syntax: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...
Under X11, copy something to your clipboard using the best tool available: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...
Here’s one I wrote recently to take a list of domains and enumerate subdomains using sublist3r. Mostly it wraps the latter tool and cleans up the output, but it also enriches the output with dig info.
https://github.com/ericfitz/dominfo
Dependencies:
Recursive grep: For every time you know you’ve written that code before but can’t remember the exact syntax. Filters out known build directories that would otherwise make it slow (moddify this to your personal use case).
https://github.com/djsamseng/cheat_sheet/blob/main/grep_for_...
#!/bin/bash
if [ $# -eq 0 ]
Seems like ripgrep would be the optimal tool for this job.
https://github.com/BurntSushi/ripgrep
A shebang-friendly script for "interpreting" single C99, C11, and C++ files, including rcfile support: https://github.com/RhysU
Use gnuplot to plot one or more files directly from the command line: https://github.com/RhysU/gplot
My workstation setup, both for Linux and MacOS, is in the following repository: https://github.com/sirikon/workstation
https://github.com/sirikon/workstation/blob/master/src/cli/c...
For Linux, it can install and configure everything I need when launched on a clean Debian installation. apt repositories, pins and packages; X11, i3, networking, terminal, symlinking configuration of many programs to Dropbox or the repository itself... The idea is to have my whole setup with a single command.
For Mac, it installs programs using brew and sets some configurations. Mac isn't my daily driver so the scripts aren't as complete.
Also there are scripts for the terminal to make my life easier. Random stuff like killing any gradle process in the background, upgrading programs that aren't packetized on APT, backing up savegames from my Anbernic, etc. https://github.com/sirikon/workstation/tree/master/src/shell
And more programs for common use, like screenshots, copying Stripe test cards into the clipboard, launching android emulators without opening Android Studio, etc. https://github.com/sirikon/workstation/tree/master/src/bin
A bloated script to automate creation of an Arch Linux Qemu VM. The subscript that runs in the VM is useful by itself for setting up a new Arch installation.
https://github.com/trevorgross/installarch/blob/main/install...
It's a personal tool that just kept growing. Probably amateurish by HN standards, but then, I'm an amateur. Yes, I could simply copy a disk image, but that's no fun.
Since everyone here like scripting, May I suggest, if you have not used it already, checkout Xbar (https://xbarapp.com/) for Mac and Argos (https://argos-scripts.github.io/) for Linux.
I have used these 2 on my machines for the last 4 years and writing tons of script for myself, here are a few:
I switched from Xero to plaintext accounting,[0] and it was a huge step up.
There are several different plaintext accounting tools, but they all support automation like this. I personally use Beancount because I work best in Python.
The other huge advantage is that the "state" of your finances isn't opaque like in Xero. If you realize you've been categorizing certain transactions incorrectly in Xero, it's a hassle to go correct everything, whereas in plaintext accounting it's usually a 2-second find/replace.
The downside is that there's a steep learning curve and the documentation is kind of overwhelming, but once you learn it, it's extremely valuable.
[0] https://plaintextaccounting.org/
I've started adding some of my shorter scripts to a single repo - https://github.com/curtis86/my-scripts
Will definitely be adding more as I tidy them up! :)
Similar to most other posters, I have a dotfiles repo, most of it isn't particularly novel, but I have a light wrapper around `git` that after a successful clone, will add custom identity information to `.git/config` so when I commit, I won't inadvertently use my work author string vs my personal author string:
https://github.com/dom111/dotfiles/blob/master/bin/git
which when combined with files like:
$cat ~/.gitidentities/github.com
Not a script as such, but I did put this together, building on what someone else did:
https://github.com/ianmiell/bash-template
It's a 'cut and paste' starter for shell scripts that tries to be as robust as possible while not going crazy with the scaffolding. Useful for "I want to quickly cut a script and put it into our source but don't want it to look totally hacky" situations.
Few years ago, I needed a quick way to create Qemu VM's locally for testing some weird software configurations. So I made a script to pull Ubuntu cloud images and clone them into qcow2 disks, then create and register libvirt virtual machines. Part of the "magic" was creating a cloud-config ISO image that would be mounted to pre-seed the VM on first launch. It also pushed my ssh key into the VM so I wouldn't need to use passwords. Janky, but worked well for what I needed.
https://github.com/noahbailey/kvmgr/blob/master/kvmgr.sh
I can only share a part, since the majority of my scripts reveal much about my system structure (I try to open whatever I can, though; the tedious part of open sourcing a script, is to make it generic/configurable).
There you go: https://github.com/64kramsystem/openscripts.
I have a nice little script for managing "MISC" packages: Manually Installed or Source Compiled.
https://github.com/tpapastylianou/misc-updater
In full honesty, I'm as proud of the "MISC" acronym as of the script itself. I'm secretly hoping the acronym catches on for referring to any stuff outside the control of a system's standard package management.
https://github.com/johnl-m/display-brightness-scripts
Nothing too complicated here, but I wanted to control my display’s brightness using my keyboard on Linux. Turned out to be pretty easy with ddcutil!
There's also modd[0] which allows for many file watch pattern -> command combos to easily be defined & run simultaneously from a modd.conf file.
[0]https://github.com/cortesi/modd
I use a script called `shell-safe-rm` [1], aliased as `rm` in interactive shells, such that I don't normally use `rm` directly. Instead of directly removing files, they are placed in the trash folder so they can be recovered if they were mistakenly deleted. Highly recommend using a script/program like this to help prevent accidental data loss.
[1] https://github.com/kaelzhang/shell-safe-rm
Somewhat boring, but I wrote a shell script to tar and gzip my home directory and then rsync it to a NAS drive.
It can be configured to exclude certain directories (.cache and Downloads being likely contenders). Also, it can read in config files so it can backup other directories.
https://github.com/lordfeck/feckback
Here's a small script I use often to tag commits with Git.
It shows the current status, lists out the most recent tags, prompts for a new tab and message, and finally pushes.
Everything is colorized so it's easy to read and I use it quite often for Golang projects.
https://github.com/bbkane/dotfiles/blob/e30c12c11a61ccc758f7...
I'm not sure bashrc tweaks completely qualify but considering it involves probably the most convoluted shell script I've ever had to come up with I'll plug https://github.com/jkern888/bash-dir-collapse. I like having the current directory in my prompt but got annoyed at how long it could get so made this to shrink the overall length without completely sacrificing the full path
Here is my script gfm-preview [1], which I think is pretty cool since it implements a HTTP server in 50 lines of shell script (ab-)use with netcat. What is does is it starts a HTTP server that serves a rendered preview of a Markdown document using GitHub's API for rendering GitHub Flavoured Markdown. The page will automatically update when the document changes using fswatch and HTTP long polling!
[1]: https://github.com/axelf4/nixos-config/blob/e90e897243e1d135...
borg-backup.sh, which runs my remote borg backups off a cronjob: https://github.com/Freaky/borg-backup.sh
zfsnapr, a ZFS recursive snapshot mounter - I run borg-backup.sh using this to make consistent backups: https://github.com/Freaky/zfsnapr
mkjail, an automatic minimal FreeBSD chroot environment builder: https://github.com/Freaky/mkjail
run-one, a clone of the Ubuntu scripts of the same name, which provides a slightly friendlier alternative to running commands with flock/lockf: https://github.com/Freaky/run-one
borg-backup.sh, which runs my remote borg backups off a cronjob: https://github.com/Freaky/borg-backup.sh
zfsnapr, a ZFS recursive snapshot mounter - I run borg-backup.sh using this to make consistent backups: https://github.com/Freaky/zfsnapr
mkjail, an automatic minimal FreeBSD chroot environment builder: https://github.com/Freaky/mkjail
run-one, a clone of the Ubuntu scripts of the same name, which provides a slightly friendlier alternative to running commands with flock/lockf: https://github.com/Freaky/run-one
borg-backup.sh, which runs my remote borg backups off a cronjob: https://github.com/Freaky/borg-backup.sh
zfsnapr, a ZFS recursive snapshot mounter - I run borg-backup.sh using this to make consistent backups: https://github.com/Freaky/zfsnapr
mkjail, an automatic minimal FreeBSD chroot environment builder: https://github.com/Freaky/mkjail
run-one, a clone of the Ubuntu scripts of the same name, which provides a slightly friendlier alternative to running commands with flock/lockf: https://github.com/Freaky/run-one
borg-backup.sh, which runs my remote borg backups off a cronjob: https://github.com/Freaky/borg-backup.sh
zfsnapr, a ZFS recursive snapshot mounter - I run borg-backup.sh using this to make consistent backups: https://github.com/Freaky/zfsnapr
mkjail, an automatic minimal FreeBSD chroot environment builder: https://github.com/Freaky/mkjail
run-one, a clone of the Ubuntu scripts of the same name, which provides a slightly friendlier alternative to running commands with flock/lockf: https://github.com/Freaky/run-one
Oh gosh, that made me soo nostalgic - I did such mono .sh file to boost my system setup couple years ago as elementaryOS at the time was not supporting updates and I had to do a clean install when new release was published: https://github.com/mrmnmly/linux-installation-script/blob/ma...
When I read it today I miss those soo oversimplified solutions to do stuff :'-)
Not a script for a specific need, but I have a folder with Bash snippets from where I copy and mix parts of them when writing scripts.
https://github.com/j1elo/shell-snippets
Mint recently updated the API they use behind the scenes, and it broke the preexisting scripts others had written for importing transactions from a CSV (when you link a new bank, it only goes up to the past 90 days).
With my son having opened an account over a year ago, but we didn’t sign up for Mint until this weekend, I ended up writing a new import script for the updated API:
https://github.com/jeradrose/mint-simple-import
Here's one I wrote a few years back, that I'm quite fond of. It turns any arbitrary directory tree with individual executables into a "git [X] [Y]" style shell command.
https://github.com/Mister-Meeseeks/subcmd/blob/master/subcmd
Have you checked out https://github.com/git-duet/git-duet/ ?
You configure a ~/.git-authors file with people with whom you regularly pair, and use `git duet [author-1] [author-2]` to set primary and secondary commit authors. Env variables set whether you want `Signed-off-by` or `Co-authored-by` trailers.
Not mine, and not ..... really.... serious.... but someone has to mention the greatest work scripts ever :
https://github.com/NARKOZ/hacker-scripts
No, but thanks for pointing out its existence. Homepage:
https://beyondgrep.com/
I started - but rarely update and kinda forgot pushing to github - some small scripts and knowledge snippets. One of them being a network/ssh based distributed unseal mechanism (using shamir algorithm) to allow machines to boot and decrypt their OS partition.
https://github.com/maulware/maulstuff
I checked in my zotero folder and wrote scripts to do daily commits and pushes to a remote host: https://github.com/jcuenod/zotero-backup-scripts/
Does my .bash_history[1] qualify?
[1] https://gitlab.com/victor-engmark/tilde/-/blob/master/.bash_...
I use git alias scripts, such as for shortcuts, metrics, and workflows.
https://github.com/gitalias/gitalias
System setup and scripts:
https://github.com/mrichtarsky/linux-shared
The repo name is a bit outdated, it works on macOS too. Lots of scripts are missing, will add them soon.
GitLab team member here, thanks for sharing!
You can also set Git push options understood by the GitLab server to create merge requests [0] on the CLI.
Sid's dotfiles provide an example in [1]. The workflow is 1) push 2) create merge request 3) set target (master/main) 4) merge when the pipeline succeeds.
alias mwps='git push -u origin -o merge_request.create -o merge_request.target=main -o merge_request.merge_when_pipeline_succeeds' # mwps NAME_OF_BRANCH
There are more push options, such as setting the MR as draft, add labels, milestones, assignees, etc. My personal favorite: Remove the source branch when the MR is merged. That's a project setting too, but sometimes not set. Using the push options, you can force this behavior and avoid stale Git branches.
glab as CLI tool provides a similar functionality to create an MR. Its development has been moved to this project [2]
[0] https://docs.gitlab.com/ee/user/project/push_options.html#pu...
[1] https://gitlab.com/sytses/dotfiles/-/blob/master/git/aliases...
[2] https://gitlab.com/gitlab-org/cli
GitLab team member here, thanks for sharing!
You can also set Git push options understood by the GitLab server to create merge requests [0] on the CLI.
Sid's dotfiles provide an example in [1]. The workflow is 1) push 2) create merge request 3) set target (master/main) 4) merge when the pipeline succeeds.
alias mwps='git push -u origin -o merge_request.create -o merge_request.target=main -o merge_request.merge_when_pipeline_succeeds' # mwps NAME_OF_BRANCH
There are more push options, such as setting the MR as draft, add labels, milestones, assignees, etc. My personal favorite: Remove the source branch when the MR is merged. That's a project setting too, but sometimes not set. Using the push options, you can force this behavior and avoid stale Git branches.
glab as CLI tool provides a similar functionality to create an MR. Its development has been moved to this project [2]
[0] https://docs.gitlab.com/ee/user/project/push_options.html#pu...
[1] https://gitlab.com/sytses/dotfiles/-/blob/master/git/aliases...
[2] https://gitlab.com/gitlab-org/cli
Also here's a emacs lisp snippet for window switching that I never managed to publish but I just can't use emacs without it. With it I press super+left and super+right to cycle windows such that if I'm on a non-asterisk buffer, it cycles only through such kind of buffer (and scratch), but if I'm on an asterisk* buffer (except scratch), it cycles only through asterisk buffers, skipping some bad buffers. And super+up goes to a non-asterisk buffer, and super+down goes to an asterisk buffer:
https://github.com/dlight/dotemacs/blob/master/bindings.el#L...
This is brittle-looking but I didn't know how to test if a buffer is special other than checking the buffer name (I'm sure there's a better way). Anyway I kept looking if emacs already had this, but apparently not
Maybe someone will think it is useful so:
(defun keys (a)
I totally missed that! Thanks for the additional detail.
This issue appears to have been fixed: https://github.com/python/cpython/issues/75820
My Mac OS bootstrap script to setup any new Mac from scratch. It includes niceties like moving the default screenshots from the Desktop to a more sane location, setting full disk encryption, and setting up privoxy & dnscrypt out if the box. https://github.com/james-see/fresh-mac
I use these aliases to deal with QR codes.
https://github.com/cednore/dotfiles/blob/master/.aliases#L46...
Related posts
- Maintain consistent styles for developers working across various editors
- ThePrimeagen builds a complete Neovim config from scratch
- numToStr/dotfiles: 🏡 /.dotfiles | Includes configs for neovim, tmux, zsh, alacrity, kitty, and more | Managed by GNU stow
- How to Make ZSH Start as Fast as Bash?
- Useful settings?