scroll VS zsv

It took me many years to get Scroll (https://scroll.pub/) to the point where I love it and am confident it will be the dominant language for writing going forward (replacing markdown).

I first had to invent Tree Notation (2017), which I got wrong on my first two tries (2012's Note and 2013's Space). Then I needed to invent Grammar (2017), and then I made the predecessor to Scroll called Dumbdown (2019). 2 years after that I shipped the first version of Scroll (2021).

Now we are on Scroll version 58 and it's blazing fast, very simple, extremely extendible, and scales very well.

It was 90% me for a while, but recently been very much a team effort.

It took a while to get right because it's a whole new kind of language, so there were a lot of mistakes that I made and had to undo, and it took a while to figure out exactly what was special about it and how to double down on that.

I recently started 2 new newspapers: https://longbeach.pub/ and http://hawaii.pub/. Very different from traditional newspapers in that they are: public domain, open source (view source on every page), and built using a new language (https://scroll.pub/).

Another cool site I found recently (via the replit guy) is https://www.rootclaim.com/

Very cool way to present arguments.

I'm thinking of taking that, as well as argdown, and building some easy to use keywords in scroll https://scroll.pub/

> Context-free grammars, and their associated parsing techniques, don't align well with real-world compilers, and thus we should deemphasise CFGs (Context-Free Grammars) and their associated parsing algorithms.

I think CFG are highly overrated. Top down recursive descent parsers are simple and allow you to craft more human languages. I think building top down parsers is something every dev should do. It's a simple technique with tremendous power.

I think the source code for Scroll (https://github.com/breck7/scroll/tree/main/grammar) demonstrates how liberating moving away from CFGs can be. Easy to extend, compose, build new backends, debug, et cetera. Parser, compiler, and interpreter for each node all in one place. Swap nodes around between languages. Great evolutionary characteristics.

I'll stop there (realizing I need to improve the docs and write a blog post).

- Uses a new language called Scroll: https://scroll.pub/

Also, put down Markdown and give our Scroll a try: https://scroll.pub

It now powers sites like my own blog (https://breckyunits.com/), knowledge bases like PLDB.com, and our first new public domain daily newspaper called the Long Beach Pub (https://longbeach.pub/1-3-2023.html).

> Java, Go, Javascript, Rust, etc are all regularly written with whitespace, and have tools to enforce such formatting, but they don't derive information from it.

Ah you reminded me. A curious phenomenon I've observed with Prettier in JS and fmt in Go is languages are moving to standardized whitespace, but as you said, not yet deriving information from it. I don't know enough about Java or Rust but I suspect they probably both have adopted a Prettier/fmt like convention where all code is formatted on save. So it seems like we are moving to a world where it will be a simple flip of a switch to then start having popular languages extract meaning from the whitespace.

> Also, Python has existed for decades and still there is little further adoption of indentation-sensitivity. It doesn't seem like a wave of indentation-sensitive languages will be coming any time soon.

I think it's coming big time this year. I think our Scroll (https://scroll.pub/) will catch fire and be the go to language instead of Markdown by the end of the year. Then with the increasing success of TreeBase (powering PLDB and others) we will start to see JSON fall for config formats and document storage databases. A lot more will happen to, data vis will be a big one, but those 2 I'm reasonably certain of happening in 2023.

GoAccess: https://goaccess.io/. I don't miss Google Analytics at all.

Loom. It's not open source I don't think but I'm digging it and excited when a public domain competitor comes out.

Our https://scroll.pub/. It's far beyond markdown at this point. I am able to not only write better but also maintain thousands of pages of content by hand (well, most of the credit for that belongs to Apple M1s, Sublime Text, git, MacOS, and Github). The stuff we are doing with it now would just not be possible with anything else, and what we're coming out with next year is super exciting. It's all public domain.

If it could be tabular in nature, maybe convert to sqlite3 so you can make use of indexing, or CSV to make use of high-performance tools like xsv or zsv (the latter of which I'm an author).

https://github.com/BurntSushi/xsv

https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql...

Parsing CSV doesn't have to be slow if you use something like xsv or zsv (https://github.com/liquidaty/zsv) (disclaimer: I'm an author). The speed of CSV parsers is fast enough that unless you are doing something ultra-trivial such as "count rows", your bottleneck will be elsewhere.

The benefits of CSV are:

- human readable

- does not need to be typed (sometimes, data in the raw such as date-formatted data is not amenable to typing without introducing a pre-processing layer that gets you further from the original data)

- accessible to anyone: you don't need to be a data person to dbl-click and open in Excel or similar

The main drawback is that if your data is already typed, CSV does not communicate what the type is. You can alleviate this through various approaches such as is described at https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql..., though I wouldn't disagree that if you can be assured that your starting data conforms to non-text data types, there are probably better formats than CSV.

The main benefit of Arrow, IMHO, is less as a format for transmitting / communicating but rather as a format for data at rest, that would benefit from having higher performance column-based read and compression

I wanted so much to use csvkit and all the features it had, but its horrendous performance made it unscalable and therefore the more I used it, the more technical debt I accumulated.

This was one of the reasons I wrote zsv (https://github.com/liquidaty/zsv). Maybe csvkit could incorporate the zsv engine and we could get the best of both worlds?

Examples (using majestic million csv):

---

}

```

This of course assumes that each line is a single record, so you'll need some preprocessing if your CSV might contain embedded line-ends. For the preprocessing, you can use something like the `2tsv` command of https://github.com/liquidaty/zsv (disclaimer: I'm its author), which converts CSV to TSV and replaces newline with \n.

You can also use something like `xsv split` (see https://lib.rs/crates/xsv) which frankly is probably your best option as of today (though zsv will be getting its own shard command soon)

Am looking for one-time help creating documentation (e.g. man pages, tutorials) for open source project (e.g. https://github.com/liquidaty/zsv) as well as product documentation for commercial products, but not enough need for a full-time job. Requires familiarity with, for lack of better term, data janitorial work, and preferably with methods of auto-generating documentation. Any suggestions as to forums or other ways to find folks who might fit the bill for ad-hoc or part-time work of this nature?

Nice work. I am a fan of tools like this and look forward to giving this a try.

However, in my first attempted query (version 3.1.6 on MacOS), I ran into significant performance limitations and more importantly, it did not give correct output.

In particular, running on a narrow table with 1mm rows (the same one used in the xsv examples) using the command "select country, count() from worldcitiespop_mil.csv group by country" takes 12 seconds just to get an incorrect error 'no such column: country'.

using sqlite3, it takes two seconds or so to load, and less than a second to run, and gives me the correct result.

Using https://github.com/liquidaty/zsv (disclaimer, I'm one of its authors), I get the correct results in 0.95 seconds with the one-liner `zsv sql 'select country, count() from data group by country' worldcitiespop_mil.csv`.

I look forward to trying it again sometime soon

All this banter arguing over CSV, JSON, sqlite seems unnecessary when you can just push format X through a pipe and get whichever format Y you want back out: https://github.com/liquidaty/zsv/blob/main/docs/csv_json_sql...

(disclaimer: I'm one of the zsv authors)