mu1 VS tup

I got sick of juggling code that migrated from one category to the other, so I wrote a little script that deals with chopping up a large source file into multiple TUs before feeding them to the compiler.

https://github.com/akkartik/mu1/blob/master/build2

More details: https://news.ycombinator.com/item?id=33574154#33575045

This really speaks to me. Modern software is too hard to assemble from source. If you're shipping sources, every moving part you add increases the odds of something going wrong on other people's computers.

It's worth having some skepticism of tools. By making some operations easy, tools encourage them. Build systems make it easy to bloat software. Package managers make it easy to bloat dependencies. This dynamic explains why Python in particular has such a terrible package management story. It's been around longer than Node or Rust, so if they seem better -- wait 10 years!

For many of my side projects I try to minimize moving parts for anyone (usually the '1' is literally true) who tries them out. I work in Unix, and one thing I built is a portable shell script that acts like a build system while being much more transparent about what it does: https://codeberg.org/akkartik/basic-build

When I use this script my build instructions are more verbose, but I think that's a good thing. They're more explicit for newcomers, and they also impose costs that nudge me to keep my programs minimalist.

You can see this build system evolve to add partial builds and parallel builds in one of my projects:

https://github.com/akkartik/mu1/blob/master/build0

https://github.com/akkartik/mu1/blob/master/build1

https://github.com/akkartik/mu1/blob/master/build2

https://github.com/akkartik/mu1/blob/master/build3

https://github.com/akkartik/mu1/blob/master/build4

Each of these does the same thing for this one repo -- build it -- but adding successively more bells and whistles.

I think providing just the most advanced version, build4, would do my users a disservice. It's also the most likely to break, where build0 is rock solid. If my builds do break for someone, they can poke around and downgrade to a simpler version.

Totally agreed!

Here's a prototype from a few years ago where I tried to make this easier: https://github.com/akkartik/mu1#readme (read just the first few paragraphs)

I still think the full answer lies in this direction.

Whenever looking at one these, I think back to the obscure but interesting "tup":

“How is it so awesome? In a typical build system, the dependency arrows go down. Although this is the way they would naturally go due to gravity, it is unfortunately also where the enemy's gate is. This makes it very inefficient and unfriendly. In tup, the arrows go up.”

https://gittup.org/tup/

Once upon a time, you could roll your own of this using `tup` which might have my favorite "how it works" in the readme:

How is it so awesome?

In a typical build system, the dependency arrows go down. Although this is the way they would naturally go due to gravity, it is unfortunately also where the enemy's gate is. This makes it very inefficient and unfriendly. In tup, the arrows go up. This is obviously true because it rhymes. See how the dependencies differ in make and tup:

[ Make vs. Tup ]

See the difference? The arrows go up. This makes it very fast.

https://gittup.org/tup/

Also has a whitepaper: https://gittup.org/tup/build_system_rules_and_algorithms.pdf

Ten years ago, I used reStructuredText and its support for LaTeX math and syntax highlighting. I used tup (tup monitor -a -f) to take care of running rst2html on save.

I might be showing my ignorance here, but this just sounds like Tup? https://gittup.org/tup/

I agree. While I like the idea of tup (https://gittup.org/tup/ -- the first "forward" build system I remember hearing of), writing a makefile is easy enough that thinking about the problem upside-down doesn't offer a compelling reason to switch.

Ptrace is one option for tracing dependencies, but it comes with a performance hit. A low-level alternative would be ftrace (https://lwn.net/Articles/608497/) or dtrace (https://en.wikipedia.org/wiki/DTrace).

Tup uses LD_PRELOAD (or equivalent) to intercept calls to C file i/o functions. On OSX it looks DYLD_INSERT_LIBRARIES would be the equivalent.

* order-only prerequisites - X must happen before Y if it's happening but a change in X doesn't trigger Y

This is just a small selection and there are missing things (like how to handle rules that affect multiple targets).

It's all horrible and complex because like a lot of languages there's a manual listing the features but not much in the way of motivations for how or why you'd use them so you have to find that out by painful experience.

It's also very difficult to address the warts and problems in (GNU) make because it's so critical to the build systems of so many packages that any breaking change could end up being a disaster for 1000s of packages used in your favorite linux distribution or even bits of Android and so on.

So it's in a very constrained situation BECAUSE of it's "popularity".

Make is also not a good way to logically describe your build/work - something like Meson would be better - where you can describe on the one hand what a "program" model was as a kind of class or interface and on the other an implementation of the many nasty operating system specific details of how to build an item of that class or type.

Make has so many complex possible ways of operating (sometimes not all needed) that it can be hard to think about.

The things that Make can do end up slowing it down as a parser such that for large builds the time to parse the makefile becomes significant.

Make uses a dependency tree - when builds get large one starts to want an Inverted Dependency Tree. i.e. instead of working out what the aim of the build is and therefore what subcomponents need to be checked for changes we start with what changed and that gives us a list of actions that have to be taken. This sidesteps parsing of a huge makefile with a lot of build information in it that is mostly not relevant at all to the things that have changed. TUP is the first tool I know about that used this approach and having been burned hard by make and ninja when it comes to parsing huge makefiles (ninja is better but still slow) I think TUP's answer is the best https://gittup.org/tup/

You might enjoy Tup[1] if you've not checked it out before.

[1]: https://gittup.org/tup/