data_jd VS Singeli

I suspect the J package Jd is probably the most non-trivial public codebase. I don’t love the coding style (functions are long and scripted) and it doesn’t make use of newer lambda functions (“direct definitions”) which are easier to read. https://github.com/jsoftware/data_jd

An example J file because this link doesn't say much:

https://github.com/jsoftware/data_jd/blob/master/csv/csv.ijs

Here's one of the more central files that ties into how a Jd database is laid out:

https://github.com/jsoftware/data_jd/blob/master/base/common...

Not that I claim anyone in particular can read it of course. Jd uses a hierarchy of folder, database, table, column that's handled with an object system to share code between them. A folder is just a place to put databases and hardly needs to add anything, while the other levels have a lot of extra functionality. As an inverted database, Jd stores each column in a file, and accesses it using memory mapping.

https://github.com/jsoftware/data_jd/blob/master/base/folder...

https://github.com/jsoftware/data_jd/blob/master/base/table....

(I designed this system when I did some of the early work to turn JDB into Jd as a summer intern)

I found this license for jd itself. It is free only for non-commercial use:

https://github.com/jsoftware/data_jd/blob/master/doc/License...

The link you mentioned only applies to the jsource folder: the jengine code.

A longer piece from GitHub: csvreportsummary=: 3 : 0 t=. <;.2 fread PATHLOGLOGFILE b=. (<,LF)=t b=. b+.(<'!')={.each t b=. b+.(<'src: ')=5{.each t b=. b+.(<'snk: ')=5{.each t b=. b+.(<'elapsed: ')=9{.each t b=. b+.(<'rows: ')=6{.each t b=. b+.(<'error: ')=7{.each t ;b#t )

Here's an AVX-2 implementation that assumes it can read up to 31 bytes past the end of the input: https://godbolt.org/z/P7PP1MnK7

Requires -fno-unroll-loops as otherwise clang gets overly unroll-y; the code is fast enough. Tail is dealt with by blending the originally read value with the new one.

(yes, that's autogenerated; from some https://github.com/mlochbaum/singeli code)

It's not ideal, but I've done this in BQN and it took about 15 lines. I didn't need to handle comments or escapes, which would add a little complexity. See functions ParseXml and ParseAttr here: https://github.com/mlochbaum/Singeli/blob/master/data/iintri...

XML is particularly simple though, dealing with something like JPEG would be an entirely different experience.