go-mysql-server VS mergestat-lite

We implemented a query optimizer with a flexible intermediate representation in pure Go:

https://github.com/dolthub/go-mysql-server

Getting the IR correct so that it's both easy to use and flexible enough to be useful is a really interesting design challenge. Our primary abstraction in the query plan is called a Node, and is way more general than the IR type described in the article from OP. This has probably hurt us: we only recently separated the responsibility to fetch rows into its own part of the runtime, out of the IR -- originally row fetching was coupled to the Node type directly.

This is also the query engine that Dolt uses:

https://github.com/dolthub/dolt

But it has a plug-in architecture, so you can use the engine on any data source that implements a handful of Go interface.

Might be interested in https://github.com/dolthub/go-mysql-server, which also does this

https://github.com/dolthub/go-mysql-server

One item under "Scope of this project":

Provide a runnable server speaking the MySQL wire protocol, connected to data sources of your choice.

I just wanted to say thanks for https://github.com/dolthub/go-mysql-server

This is incredibly useful for anyone who wants to build their own DB or wrap another datasource so it's queryable via MySQL protocol.

a very cool project they also maintain is a MySQL server framework for arbitrary backends (in Go): https://github.com/dolthub/go-mysql-server

You can define a "virtual" table (schema, how to retrieve rows/columns) and then a MySQL client can connect and execute arbitrary queries on your table (which could just be an API or other source)

Thanks for this write up! I've been really interested in postgres compatibility in the context of a tool I maintain (https://github.com/mergestat/mergestat) that uses SQLite. I've been looking for a way to expose the SQLite capabilities over a more commonly used wire-protocol like postgres (or mysql) so that existing BI and visualization tools can access the data.

This project is an interesting one: https://github.com/dolthub/go-mysql-server that provides a MySQL interface (wire and SQL) to arbitrary "backends" implemented in go.

It's really interesting how compatibility with existing protocols has become an important feature of new databases - there's so much existing tooling that already speaks postgres (or mysql), being able to leverage that is a huge advantage IMO

You can query git with this: https://github.com/mergestat/mergestat if you like the idea.

Hey, author here, happy to answer any questions! Also checkout this notebook for a deeper dive into sqlite-lines, along with a slick WASM demonstration and more thoughts on the codebase itself https://observablehq.com/@asg017/introducing-sqlite-lines

I really dig SQLite, and I believe SQLite extensions will push it to another level. I rarely reach for Pandas or other "traditional" tools and query languages, and instead opt for plain ol' SQLite and other extensions. As a shameless plug, I recently started a blog series on SQLite and related tools and extensions if you want to learn more! Next week I'll be publishing more SQLite extensions for parsing HTML + making HTTP requests https://observablehq.com/@asg017/a-new-sqlite-blog-series

A few other SQLite extensions:

- xlite, for reading Excel files, in Rust https://github.com/x2bool/xlite

- sqlean, several small SQLite extensions in C https://github.com/nalgeon/sqlean

- mergestat, several SQLite extensions for developers (mainly Github's API) in Go https://github.com/mergestat/mergestat

mergestat is an open-source tool that allows users to run SQL queries on the contents and history of git repositories.

Thanks for this write up! I've been really interested in postgres compatibility in the context of a tool I maintain (https://github.com/mergestat/mergestat) that uses SQLite. I've been looking for a way to expose the SQLite capabilities over a more commonly used wire-protocol like postgres (or mysql) so that existing BI and visualization tools can access the data.

This project is an interesting one: https://github.com/dolthub/go-mysql-server that provides a MySQL interface (wire and SQL) to arbitrary "backends" implemented in go.

It's really interesting how compatibility with existing protocols has become an important feature of new databases - there's so much existing tooling that already speaks postgres (or mysql), being able to leverage that is a huge advantage IMO

timediff is a Go package for printing human readable, relative time differences. Output is based on ranges defined in the Day.js JavaScript library, and can be customized if needed. It's currently used by the mergestat command-line interface.

Assuming committers adhere to it, there could be some interesting use cases when combined with a tool like AskGit (https://github.com/askgitdev/askgit) for understanding what "categories" of work is being done in a codebase.

Maybe even what directories/files tend to see `fix` or `refactor` more frequently (signs of a poorly design or "hot" area?)

I've been very curious to explore this type of use case with askgit (https://github.com/augmentable-dev/askgit) which was designed for running simple "slice and dice" queries and aggregations on git history (and change stats) for basic analytical purposes. I've been curious about how this could be applied to a small text+git based "db". Say, for a regular json or CSV dumps.

This also reminds me of Dolt: https://github.com/dolthub/dolt which I believe has been on HN a couple times