resholve
forkrun
Our great sponsors
resholve | forkrun | |
---|---|---|
11 | 18 | |
208 | 136 | |
- | - | |
7.7 | 9.7 | |
6 days ago | 7 days ago | |
Python | Shell | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
resholve
-
What is the Flakes version of "reproducible interpreted scripts"?
I'm also not 100% on whether it answers the question, but I imagine you're thinking of https://github.com/abathur/resholve (doc in nixpkgs: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/misc/resholve/README.md)
-
modular bash profile scripting with shellswain
I intend to eventually find some time to figure out how feasible it would be to use https://github.com/abathur/resholve or wrapper techniques to bolt basalt (and perhaps other bash PMs) on to the nix ecosystem and nix-package some of your libraries.
-
Is there a good way to programmatically determine how many inputs some function can support?
(I'd love to have this ability for https://github.com/abathur/resholve to reliably identify arguments to one command that are also external commands/programs that it will in turn exec. I can't imagine trying to start it until/unless I have any bright ideas about how that executable spec and a parser for it would work.)
-
Could someone give me an example how I would have multiple "commands" in default.nix?
In https://github.com/abathur/resholve/blob/master/default.nix and https://github.com/abathur/resholve/blob/master/shell.nix you can see one approach to extending that line of thought to the default.nix itself.
-
Building the Future of the Command Line
Completions have in general been of interest, though the shell-specific completions I've looked at so far were all too dynamic.
I'd forgotten all about Fig since I saw your launch post here last year, so thanks for reminder. (I don't think I had quite started to work on parsing specific external commands, yet. Was still focused on just identifying the likely presence of exec in the executables.)
Are you familiar with the parse code? Are you handling painful stuff like combined short flags with a trailing option? (If I ferreted out some of the more painful cases I've had to wrangle, I am curious if you'd have a gut sense of whether your approach handles it. Would you mind if I reach out? I am working on this for https://github.com/abathur/resholve)
-
Devbox: Instant, easy, and predictable shells and containers
@dloreto @robrich A little aside from the announcement, but since it seems like you both work on this I wanted to surface something that came up down in a subthread:
I'm curious if you attempted to support macOS by doing this with Nix's dockerTools and cross-compiling (there may be better sources, but it's at least hinted at in https://nix.dev/tutorials/building-and-running-docker-images...)? If so, I'm wondering where that failed or bogged down?
---
Background: I build a tool (https://github.com/abathur/resholve) for ~packaging Bash/Shell (i.e., for demanding all dependencies be present). The tool's technically agnostic, but I built it specifically to fix Shell packaging in Nix.
I think it could benefit a lot of other Shell projects, since one of Shell's big tribulations is dealing with heterogenous environments, but most Shell projects wouldn't see much reason to endure the pain of adopting Nix if they still had to support the heterogenous environments.
Much like you're doing here, I've entertained figuring out how to build a Nix-based packaging flow that can generate deployable standalone bundles or containers. It'd be a heavy way to bundle Shell, but I imagine some projects would take the tradeoff for predictability and reduced support load. But since it would need to take place within a Nix build, I'd need to cross-compile for it to work on macOS. Hoping you know if it's a dead-end or not :)
-
Ask HN: Why aren't code diagram generating tools more common?
For a concrete example, I've been developing a tool (https://github.com/abathur/resholve) that can ~build/link Bash/Shell scripts--i.e., rewrite them with external executables converted to absolute paths. (This helps ensure dependencies are known, declared, present, and don't have to be on the global PATH for the script to execute cleanly.)
There's a devilish sub-problem, which is that any given executable can potentially exec arbitrary arguments. For now I handle this with a very crude automated binary/executable analysis that needs to be augmented by human source analysis. Deep multi-language source analysis wouldn't be very scalable, but I suspect fairly-standardized structural annotations could improve the results in a scalable way.
I have to imagine there are other applications of the same information.
-
I wrote an article about creating unit-tests and mocks for POSIX shells
I'm not 100% sure if you see the minimal PATH dependencies as a problem or not--so this may or may not help--but I develop https://github.com/abathur/resholve to make it easier to package Shell in Nix/nixpkgs by rewriting invocations of external dependencies in Shell scripts to absolute paths--and shunit2 is one of the Nix packages that I've updated to use resholve.
-
On Env Shebangs
I came here to say this, too :)
But, of course, it still isn't a silver bullet...
1. You still have to have a sane PATH. A fair amount of the Nix install-related issues that get opened are PATH problems, and you can also run into problems with PATH in cron/launchd.
2. You still have to know what the script depends on. This can get tricky beyond small scripts you wrote yourself. (I write a tool for ~linking/resolving external dependencies in Shell scripts, https://github.com/abathur/resholve. As I've been working on converting some of nixpkgs' existing Shell packages to use it, I almost always find dependencies the initial packager missed.)
-
Runtime dependencies for a bash script
Check out resholve. https://github.com/abathur/resholve
forkrun
- Forkrun: Runs multiple inputs through a command in parallel using bash coprocs
- Forkrun – A pure-bash function for parallelizing loops
-
rparse: an easy-to-use shell script/function option parser that uses regex to determine which inputs are options
EDIT: moved code + example to GITHUB
-
[BASH] Can I use procfs to change the file that a backgrounded process is writing data to?
One example that I dont know how to do streaming for involves breaking up the stdin stream and sending it to different places, but where you determine where to send it in realtime. This isnt the code that spawned this particular thread, but a good example of this is in this function I wrote called forkrun.
-
Optimizing bash scripts?
If you want an example of a really optimized bash script check out my forkrun utility. I spent a really long time optimizing it. It parallelizes loops (like xargs -P and parallel), and for problems with "many very quick iterations" (e.g., sha256sum of a million tiny files) it is twice as fast as the fastest method available via xargs or parallel (which means it is outpacing well optimized compiled C/C++ binaries).
-
Bash continuous parallelization
as it so happens, I recently wrote a script called forkrun that parallelizes loops really fast. In particular for things like your simulation that require running 1 at a time in parallel (N parallel batches of 1 simulation, as opposed to N parallel batches of M simulations) the parallelization framework itself is 7x faster than xargs -l 1 -P N and ~25x faster than parallel -j N.
-
Globals or not globals?
However, if you have bash 4 or later, using bash coprocs can offer much better performance. The idea is that you spawn a bunch of persistent coprocs and pipe data to them, allowing you to parallelize the loop without forking each individual iteration. I wrote / am writing a utility called forkrun that parallelizes loops using bash coprocs. You're welcome to use it, though heads up it is still in beta (almost everything works, but I am still debugging the -k flag sometimes not working properly...-k ensures the output order is the same as input order). If nothing else it gives a real world example of parallelizing using coprocs.
-
Can you force bash to not give a throw a specific error?
Side note: in case you were wondering the actual code that inspired this post is my forkrun utility. If you felt like checking it out Id love to hear any feedback you might have on it.
-
Out of curiosity, what is your best script you can showcase?
For me? My personal best script is (by a wide margin) my forkrun utility. It uses bash coprocs to efficiently parallelize loops. Usage is nearly identical to xargs -P <#>. Options are similar to xargs....a few are missing (namely those related to running things interactively), and a few options are things xargs doesnt do but forkrun can.
-
How can I run commands in parallel and write the output of each command to different linux terminals, one linux terminal for each command running in parallel.
I wrote a nifty function for parallelizing loops in bash really fast called forkrun. It will, with the help of tail -F, let you do this (among many other things) by writing the output from each parallel worker to a separate file, which you can then monitor in real time with tail -F in another terminal window.
What are some alternatives?
datashare - A self-hosted search engine for documents.
pkm - A super minimal TUI package manager wrapper written in BASH v4.2+
mpack - MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]
bin - my bins (mostly shell scripts)
egglog0 - Datalog + Egg = Good
dyetide - a bash script that replaces hex, rgb, or hsl color codes out for other color codes. either within a file or from the terminal!
devbox - Instant, easy, and predictable development environments
dun - Meeting notes and todo tasks CLI
py_regular_expressions - Learn Python Regular Expressions step by step from beginner to advanced levels
fml - :card_index_dividers: A stupid simple, fast TUI file manager written in BASH v4.2+
swift-sh - Easily script with third-party Swift dependencies.
wgs - A minimal wallpaper getter setter written in pure BASH