gojq VS brackit

Yeap i've talked to itchyny quite a lot about various changes https://github.com/itchyny/gojq/issues/153 and also upstreamed quite a lot https://github.com/itchyny/gojq/issues?q=author%3Awader like custom iterators (to allow eval, own iterators and "empty" functions), query marshalling (query rewrite tricks) and a bunch of small things and bug fixes. But the largest change to add a JQValue interface is quite complex, other changes like extended literals is also a bit tricky.

Hmm weird list of changes for https://github.com/wader/gojq/compare/fq...itchyny:gojq:main but i guess it is because i haven't kept my main branch in sync. The fq branch should be based on latest gojq/main as of now. I usually try to rebase as quick as possible.

Let me know if you have any other questions or want to help out! maybe email etc as i usually don't check HN comments replies that often :)

gojq has support for yaml input (via a very annoying argument name) and also has the golang property of "curl binary; chmod; profit": https://github.com/itchyny/gojq#difference-to-jq

It's error reporting is also clang-vs-gcc level wizardry, and I often use it to get a helpful message instead of "ENOWORKY" from jq (I haven't tried 1.7 yet, so it could be better for all I know)

Some competition for https://github.com/itchyny/gojq. I had read somewhere that it was faster than jq - no idea if that's still the case.

JQ has a go implementation usable as a library I see. The project looks fairly active https://github.com/itchyny/gojq

I've been getting a lot of mileage out of https://github.com/itchyny/gojq#readme recently due to two things: its vastly superior error messages and the (regrettably verbose) `--yaml-input` option

I also have https://github.com/01mf02/jaq#readme installed but just haven't needed it

I use gojq with --yaml-input or --yaml-output and flip back and forth between JSON and YAML promiscuously and have 100% jq UI compat, which helps because I use jq a lot. First thing I looked at on yq is '-s', which is 'slurp' for jq. Slightly altered semantics would just trip me up, and it seems like you can make a nearly straight bijection between YAML and jq so you can just do exactly the same things with either one (with some minor exceptions.)

https://github.com/itchyny/gojq

jq https://github.com/itchyny/gojq

github.com/itchyny/gojq -> similar to goawk, except JQ this time

I've already posted the project a couple of years ago and it gained some interest, but a lot of stuff has been done since then, especially regarding performance, a complete new JSON store, a REST API, various internals refactored, an improved JSONiq based query engine allowing updates, a now already dated web UI, a new Kotlin based CLI, a Python and TypeScript client to ease the use of Sirix...

First prototypes from a precursor stem already from 2005.

So, what is it all about?

I'm working on an evolutionary data store in my spare time[1]. It is based on the idea to get rid of the need for a second trx log (the WAL) by using a persistent tree of tries (preserving the previous revision through copy on write and path copying to the root) index as the log itself with only a single permitted read/write txn concurrently and in parallel to N read-only txns, which are bound to specific revisions during the start. The single writer is permitted on a resource (comparable to a table/relation in a relational DB) basis within a database, reads do not involve any locks at all.

The idea is, that the system atomically swaps the tree root to the new version (replicated). If something fails the log can simply be truncated to the former tree root.

Thus, the system has many similarities with Git (structural sharing of unchanged nodes/pages) and ZFS snapshots (regarding the latter the keyed trie has been inspired by ZFS, as well as that checksums for child pages are stored in parent pages in the references to the child pages)[2].

You can of course simply execute time travel queries on the whole revision history, add commit comments and the author to answer questions such as who committed what at which point in time and why...

The system not only copies full data pages, but it applies a sliding snapshot versioning algorithm to keep storage space to a minimum.

Thus, it's best suited for fast flash drives with fast random reads and sequential writes. Data is never overwritten, thus audit trails are given for free.

The system stores find granular JSON nodes, thus the structure and size of an object has almost no limits. A path summary is built, which is an unordered set of all paths to leaf nodes in the tree and enables various optimizations. Furthermore a rolling hash is optionally built, whereas during inserts all ancestor node hashes are adapted.

Furthermore it optionally keeps track of update operations and the ctx nodes involved during txn commits. Thus, you can easily get the changes between revisions, you can check the full history of nodes, as well as navigate in time to the first revision, the last revision, the next and previous revision of a node...

You can also open a revision at a specific system time revert to a revision and commit a new version while preserving all revisions in-between.

As said one feature is, that the objects can be arbitrarily nested, thus almost no limits in the number and updates are cheap.

A dated Jupyter notebook with some examples can be found in [3] and overall documentation in [4].

The query engine[5] Brackit is retargetable (a couple of interfaces and rewrite rules have to be implemented for DB systems) and especially finds implicit joins and applies known algorithms from the relational DB systems world to optimize joins and aggregate functions due to set-oriented processing of the operators.[6]

I've given an interview in [7], but I'm usually very nervous, so don't judge too harshly.

Give it a try and happy coding!

Kind regards

Johannes

[1] https://sirix.io | https://github.com/sirixdb/sirix

[2] https://sirix.io/docs/concepts.html

[3] https://colab.research.google.com/drive/1NNn1nwSbK6hAekzo1YbED52RI3NMqqbG#scrollTo=CBWQIvc0Ov3P

[4] https://sirix.io/docs/

[5] http://brackit.io

[6] https://colab.research.google.com/drive/19eC-UfJVm_gCjY--koOWN50sgiFa5hSC

[7] https://youtu.be/Ee-5ruydgqo?si=Ift73d49w84RJWb2

Append-only database system (based on a persistent inddx structure): https://github.com/sirixdb/sirix or a retargetable query compiler https://github.com/sirixdb/brackit

[2] http://brackit.io

Hello,

I want to find enthusiastic OSS frontend developers for my JSON data store project[1], which is able to retain the full revision history of a database resource (binary JSON) through small sized copy-on-write snapshots of the main index tree of tries and a novel sliding snapshot algorithm.

As I'm a fan of compilers (http://brackit.io) I think either working on the current frontend with Svelte[2], which is currently really dated and uses Sapper or a new frontend using SolidJS would be great.

What are the advantages/disadvantages of both frameworks in your opinion? I'm a backend software engineer, but maybe SolidJS is more familiar to frontend devs because of JSX and at least in benchmarks it seems to be faster. But maybe the differences except for the different syntaxes aren't that big.

I envision visualizations for comparing revisions of resources or subtrees therein and also to visualize time travel queries. A screenshot of the old frontend: https://github.com/sirixdb/sirix/blob/master/Screenshot%20from%202020-09-28%2018-50-58.png

Let me know which framework you'd prefer for the task at hand and what are the advantages/disadvantages in your opinion for both of them in general.

If you want to help, it's even better. Let me know :-)

[1] https://sirix.io || https://github.com/sirixdb/sirix

Basically JSONiq, with a few minor syntax differences.

Our query engine/compiler is and can be used by other data stores as well:

http://brackit.io

That's one of the main steps forward for Brackit, a retargetable JSONiq query engine/compiler (http://brackit.io) and the append-only data store SirixDB (https://sirix.io) and a new web frontend. My vision is not only to explore the most recent revision but also any other older revisions, to display the diffs, to display thd results of time travel queries... help is highly welcome as I'm myself a backend engineer and working on the query engine and the data store itself :-)