Nestor Alternatives

nestor reviews and mentions

• Professor Donald Knuth on Writing and More, an Interview [pdf]
  4 projects | news.ycombinator.com | 30 Dec 2020

  (In fact his earlier non-literate TeX78 in the SAIL language was written as a bunch of separate modules rather than a single 25000-line file like the literate/"better" tex.web.)

  So that's a bit about the genesis of literate programming. Now about some of your other comments:

  - Your quote about top-down programming from literateprogramming.com is from the author of fweb, but actually Knuth himself doesn't advocate top-down programming. One of the things he likes about literate programming is that the program can be written in a mixture of bottom-up and top-down programming without being forced to commit to either; from studying many of his programs it's clear that his preference is to build things mostly bottom-up, though each small "layer" he may write top-down. (For example, in his TeX program he starts with "chapters" on string handling, arithmetic, memory allocation, data structures etc, building up to the main program as the climax of the book. Each "chapter", which is sometimes just a single function, is written top-down, more or less.)

  - You linked to how the PBRT book introduces literate programming, and said it shows noweb/LP is "still too mired by constraints of C", but it's just that the example they used for the idea of rearranging sections is one of overcoming some constraints of C using LP. (It's not really specific to C though; the same thing would apply whenever one had a lot of initialization to do in different places, like the giant object constructors one can see in many C++ codebases.)

  - Tastes are subjective, but IMO the program he wrote for Bentley's question of "top k most common words" (not his choice of problem!) is actually very interesting: it's an exposition of an ingenious data structure called a (packed) hash trie that has, to the best of my knowledge, never been described anywhere else, and which he probably made up on the spot for this problem. (Packed tries are described in an exercise in TAOCP and the TeX program uses them to store hyphenation patterns—his student Liang's thesis was about this—but packing them randomly using an open-addressing hash table seems to be a twist particular to this program.) I was planning to write a blog post last year illustrating this with pictures and all that, but never got around to it. I did write a translation into (non-literate) C++ here: https://codegolf.stackexchange.com/a/197870

  - Returning to a couple of your other comments and the phrase "top-down style associated with literate programming": I think another way of looking at this is historical. When Knuth learned to program, in machine code on an IBM 650 and later writing compilers in machine code (or with his own assemblers) other machines, even the idea of subroutines (with standard calling conventions and all that) was hardly well-established, which is why TAOCP Chapter 1 has, under 1.4 "Some Fundamental Programming Techniques" an explanation of what subroutines are. When "Structured Programming" took over in the 1970s (just the idea of using if/while/for and subroutines instead of arbitrary control-flow jumps), and even that Knuth is not still on board with. (If anyone is interested I'll write a blog post on Knuth's opinion on/defence of "goto", which he still uses.) Since then, the programming / software-engineering world at large has settled on certain ideas of structuring programs, in order to keep complexity manageable: the use of abstraction, encapsulation, modules, information-hiding, objects, avoiding global variable,s etc. In some sense what Knuth came up with for himself with WEB/CWEB is a wholesale alternative to all these ways of structuring programs. This seems to suffice for him, but for the rest of us it's a highly unfamiliar/unpalatable way of writing programs. This doesn't mean "Knuth is doing it wrong" in the sense of Kartik's post, because what you should take away as "Literate Programming" from reading his programs is not the diff w.r.t. how a modern programmer would it, but the diff w.r.t. how he would write it without LP. That is, if you take for granted that the program is going to have very few functions, lots of global variables, etc., the question is whether LP is a better way of writing or presenting that program than non-LP. (It seems yes, though even this is debatable. Besides, Knuth does say that without LP he'd probably write more levels of functions but with LP he doesn't "have to", so you may disagree with my assertion too.) You should instead look at examples of programs that a modern programmer inculturated in modern dogma wrote with and without LP. I imagine these days one might have better results searching for people using org-babel (rather than cweb/noweb), or the https://github.com/AndrewOwenMartin/nestor mentioned in another thread above, or codebase of/written in nbdev that came up recently (https://www.fast.ai/2019/12/02/nbdev/ , https://github.blog/2020-11-20-nbdev-a-literate-programming-...).

  - But that apart, returning once again to Knuth's own literate programs. I have struggled a lot with them. I'm sure there are many people who have understood more than me, but I suspect I have struggled more than anyone :D I had some grand plans to make it more readable etc (https://shreevatsa.net/tex/program), but eventually they started making more sense as they are. (I should update those pages…) The point is, literate programming does not necessarily mean writing code in a vacuum for readers who know nothing. Knuth does not imagine explaining language features (as you pointed out with the INFORM example in another comment here), and moreover he also assumes familiarity with his LP conventions (the overall structure of the program, how to use the index and flip back-and-forth) and even I think his usual programming conventions. So the fact that the ADVENTURE program has the source file's outline (with #includes and everything at the top) on the second page, with a typedef of the "boolean" type and a couple of general-purpose variables declared at the top of "main" (which will probably be (ab)used for lots of different purposes, I expect)—all of these are not a problem, for the "literate" reader. To Knuth, literate programming does not mean belabouring and explaining everything; it's just a way of structuring your program. And though he mentions reading the program like a book, he doesn't think of books as things to be read strictly word-for-word in the sequence they appear; he imagines reading any book non-linearly (looking up the index, flipping back-and-forth, skimming, etc) and that's how the programs are to be read too: the order is just more or less the "right" one, not strictly.

  About the ADVENTURE program in particular, let me post a separate comment as this is already very long.