go-mysql-server VS lakeFS

Might be interested in https://github.com/dolthub/go-mysql-server, which also does this

https://github.com/dolthub/go-mysql-server

One item under "Scope of this project":

Provide a runnable server speaking the MySQL wire protocol, connected to data sources of your choice.

I just wanted to say thanks for https://github.com/dolthub/go-mysql-server

This is incredibly useful for anyone who wants to build their own DB or wrap another datasource so it's queryable via MySQL protocol.

a very cool project they also maintain is a MySQL server framework for arbitrary backends (in Go): https://github.com/dolthub/go-mysql-server

You can define a "virtual" table (schema, how to retrieve rows/columns) and then a MySQL client can connect and execute arbitrary queries on your table (which could just be an API or other source)

Thanks for this write up! I've been really interested in postgres compatibility in the context of a tool I maintain (https://github.com/mergestat/mergestat) that uses SQLite. I've been looking for a way to expose the SQLite capabilities over a more commonly used wire-protocol like postgres (or mysql) so that existing BI and visualization tools can access the data.

This project is an interesting one: https://github.com/dolthub/go-mysql-server that provides a MySQL interface (wire and SQL) to arbitrary "backends" implemented in go.

It's really interesting how compatibility with existing protocols has become an important feature of new databases - there's so much existing tooling that already speaks postgres (or mysql), being able to leverage that is a huge advantage IMO

Might want to look at purpose built tools for that such as lakeFS (https://github.com/treeverse/lakeFS/)

* Disclaimer: I'm one of the creators/maintainers of the project.

Might want to checkout lakeFS: https://github.com/treeverse/lakeFS

(full disclosure: I'm one of the creators)

Also in the same vein, check out https://lakefs.io/

Git was designed for managing software development projects and for versioning text/code files. Therefore, Git doesn’t handle large files. Git released Git LFS (Large File System) to overcome large file versioning, which is better than Git, but fails when scaling. Also, both Git and Git LFS are not optimized for data science workflow. To overcome this challenge, many powerful tools emerged in recent years, such as DVC, Delta Lake, LakeFS, and more.

LakeFS: https://lakefs.io/

Hello HN!

We're Oz and Einat, co-founders of lakeFS (https://lakefs.io/), an open-source project that allows the creation of performant git-like repositories over an object store (i.e. S3).

Prior to starting lakeFS we were VP of R&D and CTO at SimilarWeb, a (now-public) Israeli web analytics company whose business model is based on the collection and analysis of the internet's activity.

Recovering from a pernicious error in a million S3 files shouldn't require a full day or even week of work to fix… instead let's make it an instantaneous revert operation to a previous commit.

The challenge to implement this type of functionality is a technical one, one we took it upon ourselves to solve. It's been 1 year since the first public commit on lakeFS and we've now raised a $15M Series A to continue building and improving our vision.

We've evolved a ton in the past year, completely refactoring the data model to remove dependency on Postgres. Fittingly, we now use rocksDB on the object store to persist the metadata lakeFS manages (with the added benefit of simplifying the installation process). Check out the roadmap to follow our progress on building out native integrations with other important technologies in the open data stack such as Spark, Hive Metastore, and Delta Lake.

We encourage you to check out our Github repo: (https://github.com/treeverse/lakeFS) and documentation pages: (https://docs.lakefs.io/).

We're proud of how far we've come, but know there's lots more work to do. Please do let us know your thoughts!

treeverse/lakeFS (Go): An open source platform that delivers resilience and manageability to object-storage based data lakes