neatroff VS asciidoctor-latex

I recommend sticking with Groff. There are other implementations of Troff—some older (Heirloom Doctools), some newer (Neatroff)—but most of the documents you'll encounter in the wild will have been written with Groff in mind.

Plan 9 troff might work! It works with utf8 out of the box[0], and while I haven't used it for complex math typesetting, there is a command (eqn [1]) that was developed for it. I'd recommend Ali Rudi's port (neatroff [2][3]) for a minimalist implementation. There's also Heirloom Documentation Tools [4] which is an implementation of *roff-and-friends that uses Knuth's paragraph-at-once algorithm (instead of the original line-wise one) for typesetting, plus some other interesting features.

The authors of eqn wrote a paper about it: "Typesetting Mathematics" by Brian Kernighan and Lorinda Cherry. Kernighan also wrote two manuals (one in 1976 with a revision in 1992, and one in 2007 with updates for the Plan 9 version). [5].

[0] utf8 was developed by Ken Thompson and Rob Pike during the creation of Plan9. The entire OS is compatible. Story here: http://doc.cat-v.org/bell_labs/utf-8_history

[1] http://man.cat-v.org/plan_9/1/eqn

[2] https://github.com/aligrudi/neatroff

[3] PDF manual for neatroff: http://litcave.rudi.ir/neatroff.pdf

[4] https://n-t-roff.github.io/heirloom/doctools.html

[5] These (and more) can be found here: http://www.kohala.com/start/troff/troff.html

How close does AsciiDoctor + asciidoctor-latex get to that? https://github.com/asciidoctor/asciidoctor-latex

I've only been using Asciidoctor for a few weeks but I'm already an MD convert.