Stateless VS ndjson-spec

I like this method and state machines in general. Makes it more easier to reason about complex software. Also, ithe f the state machine gets bigger (more states) in the past I have used the Stateless library which is very good. But what you described above is perfect for managing the state of a menu system.

After using stateless for three years, my conclusion is that I cannot imagine a scenario, where it makes sense to use it.

This is a great article. Here is the Stateless library he is writing about. The link to the library is kind of buried in the middle of a sentence.

Yes! And we're using one. https://github.com/dotnet-state-machine/stateless We're creating the configuration dynamically based on how the user wants his workflow to work.

Stateless is my go-to state machine library for .NET. It's reasonably advanced but can also be very simple if you don't need the more advanced features.

I'm playing around with this library for a medium sized state machine I'm setting up, and it has this functionality. I think explicit state machines are a good use-case for having an actual diagram, since they can take up enough space that it becomes hard to build a mental map from reading the code due to locality issues.

What’s the difference between terminators and separators here? The ndjson spec [0] doesn’t say anything like that, and it seems that ndjson and jsonlines are identical in what documents they accept.

[0]: https://github.com/ndjson/ndjson-spec

It's possible to work around this and still use a JSON-ish format. For example, my @broofa/persistentmap project provides an ES6 Map API backed by a persistent append-only file in NDJSON format. (Note: NDJSON isn't really a spec so much as a "convention"). But, honestly,

ndjson is actually a really pragmatic choice here that should not be overlooked.

Tabular formats break down when the data stops being tabular. This comes up a lot. People love spread sheets as editing tools but they then end up doing things like putting comma separated values in a cell. I've also seen business people use empty cells to indicate hierarchical 'inheritance". An alternate interpretation of that is that that data has some kind of hierarchy and isn't really row based. People just shoehorn all sorts of stuff into spreadsheets because they are there.

With ndjson, every line is a json object. Every cell is a named field. If you need multiple values, you can use arrays for the fields. Json has actual types (int, float, strings, boolean). So you can have both hierarchical and multivalued data in a row. The case where all the fields are simple primitives is just the simple case. It has an actual specification too: https://github.com/ndjson/ndjson-spec. I like it because I can stream process it and represent arbitrarily complex objects/documents instead of having to flatten it into columns. The parsing overhead makes it more expensive to use than tsv though. The file size is fine if you use e.g. gzip compression. It compresses really well generally.

But I also use either tab separated values quite often for simpler data. I mainly like it because google spread sheets provides that as an export option and is actually a great editor for tabular data that I can just give to non technical people.

Both file formats can be easily manipulated with command line tools (jq, csvkit, sed, etc.). Both can be processed using mature parsers in a wide range of languages. If you really want, you can edit them with simple text editors, though you probably should be careful with that. Tools like bat know how to format and highlight these files as well. Etc. Tools like that are important because you can use them and script them together rather than reinventing wheels.

Formats like parquet are cumbersome mainly because none of the tools I mention support it. No editors. Not a lot of command line tools. No formatting tools. If you want to inspect the data, you pretty much have to write a program to do it. I guess this would be fixable but people seem to be not really interested in doing that work. Parquet becomes nice when you need to process data at scale and in any case use a lot of specialized tooling and infrastructure. Not for everyone in other words.

Character encoding is not an issue with either tsv or ndjson if you simply use UTF-8, always. I see no good technical reason why you should use anything else. Anything else should be treated as a bug or legacy. Of course a lot of data has encoding issues regardless. Shit in, shit out basically. Fix it at the source, if you can.

The last point is actually key because all of the issues with e.g. csv usually start with people just using really crappy tools to produce source data. Switching to a different file format won't fix these issues since you still deal with the same crappy tools that of course do not support this file format. Anything else you could just fix to not suck to begin with. And if you do, it stops being an issue. The problem is when you can't.

Nothing wrong with tsv if you use UTF-8 and a some nice framework that generates properly escaped values and does all the right things. The worst you can say about it is that there are a bit too many choices here and people tend to improvise their own crappy data generation tools with escaping bugs and other issues. Most of the pain is self inflicted. The reason csv/tsv are popular is that you don't need a lot of frameworks / tools. But of course the flipside is that DYI leads to people introducing all sorts of unnecessary issues. Try not to do that.

Newline-separated JSON is pretty close. It's not exactly a "standard" standard like something the IETF defines, but it's pretty easy to work with anyhow.