Top 23 Go MySQL Projects
-
Project mention: Comparing Nginx Performance in Bare Metal and Virtual Environments | news.ycombinator.com | 2021-10-29
I do agree with you in that regard, however, that's also a dangerous line of thinking.
There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility
There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up.
Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal with the complexities of balancing the load and dealing with backpressure - introduce a message queue if you must, maybe even an in-memory one for simplicity's sake, or consider an event based architecture where "what needs to be done" is encapsulated within a data structure that can be passed around and applied whenever possible. In my eyes, such solutions can in many cases be better than losing the many benefits of having a single source of truth.
Alternatively, consider sharding as a possibility, or, alternatively, do some domain driven design, figure out where to draw some boundaries and split your service into multiple ones that cover the domain with which you need to work with. Then you have one DB for sales, one for account management, one for reports and so on, all separated by something as simple as REST interfaces and with rate limits or any of the other mechanisms.
If, however, neither of those two groups of approaches don't seem to be suitable for the loads that you're dealing with, then you probably have a team of very smart people and a large amount of resources to figure out what will work best.
To sum up, if there are no good solutions in the space, perhaps that's because the problems themselves haven't been solved yet. Thus, sooner or later, they'll need to be sidestepped and their impact mitigated in whatever capacity is possible.
-
GitHub pretty much did that. They use Vitess to scale MySQL.
-
Scout APM
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
-
go-sql-driver/mysql
Go MySQL Driver is a MySQL driver for Go's (golang) database/sql package (by go-sql-driver)
I'm following Alex Edwards web dev book, he uses MySQL with go-sql-driver/mysql , i want to use Postgres, which one should I use?
-
- gh-ost
I authored the original schema change tool, oak-online-alter-table https://shlomi-noach.github.io/openarkkit/oak-online-alter-t..., which is no longer supported, but thankfully I did invest some time in documenting how it works. Similarly, I co-designed and was the main author for gh-ost, https://github.com/github/gh-ost, as part of the database infrastructure team at GitHub. We developed gh-ost because the existing schema change tools could not cope with our particular workloads. Read this engineering blog: https://github.blog/2016-08-01-gh-ost-github-s-online-migrat... to get better sense of what gh-ost is and how it works. I in particular suggest reading these:
- https://github.com/github/gh-ost/blob/master/doc/cheatsheet....
- https://github.com/github/gh-ost/blob/master/doc/cut-over.md
- https://github.com/github/gh-ost/blob/master/doc/subsecond-l...
- https://github.com/github/gh-ost/blob/master/doc/throttle.md
- https://github.com/github/gh-ost/blob/master/doc/why-trigger...
At PlanetScale I also integrated VReplication into the Online DDL flow. This comment is far too short to explain how VReplication works, but thankfully we again have some docs:
- https://vitess.io/docs/user-guides/schema-changes/ddl-strate... (and really see entire page, there's comparison between the different tools)
- https://vitess.io/docs/design-docs/vreplication/
- or see this self tracking issue: https://github.com/vitessio/vitess/issues/8056#issue-8771509...
Not to leave you with only a bunch of reading material, I'll answer some questions here:
> Can you elaborate? How? Do they run on another servers? Or are they waiting on a queue change waiting to be applied? If they run on different servers, what they run there, since AFAIK the migration is only DDL, there's no data?
The way all schema change tools mentioned above work is by creating a shadow aka ghost table on the same primary server where your original table is located. By carefully both copying data from original table as well as tracking ongoing changes to the table (whether by utilizing triggers or by tailing the binary logs), and using different techniques to mitigate conflicts between the two, the tools populate the shadow table with up-to-date data from your original table.
This can take a long time, and requires an extra amount of space to accommodate the shadow table (both time and space are also required by "natural" ALTER TABLE implementations in DBs I'm aware of).
With non-trigger solutions, such as gh-ost and VReplication, the tooling have almost ocmplete control over the pace. Given load on the primary server or given increasing replication lag, they can choose to throttle or completely halt execution, to resume later on when load has subsided. We have used this technique specifically at GitHub to run the largest migrations on our busiest tables at any time of the week, including at peak traffic, and this has show to pose little to no impact to production. Again, these techniques are universally used today by almost all large scale MySQL players, including Facebook, Shopify, Slack, etc.
> who will throttle, the migration? But what is the migration? Let's use my example: a column type change requires a table rewrite. So the table rewrite will throttle, i.e. slow down? But where is this table rewrite running, on the main server (apparently not) or on a shadow server (apparently either since migrations have no data)? Actually you mention "when your production traffic gets too high". What is "high", can you quantify?
The tool (or Vitess if you will, or PlanetScale in our discussion) will throttle based on continuously collecting metrics. The single most important metric is replication lag, and we found that it predicts load more than any other matric, by far. We throttle at 1sec replication lag. A secondary metric is the number of concurrent executing threads on the primary; this is mroe improtant for pt-online-schema-change, but for gh-ost and VReplication, given their nature of single-thread writes, we found that the metric is not very important to throttle on. It is also trickier since the threshold to throttle at depends on your time of day, particular expected workload etc.
> We run customers that do dozens to thousands of transactions per second. Is this high enough?
The tooling are known to work well with these transaction rates. VReplication and gh-ost will add one more transaction at a time (well, two really, but 2nd one is book-keeping and so low volume that we can neglect it); the transactions are intentionally kept small so as to not overload the transaction log or the MVCC mechanism; rule of thumb is to only copy 100 rows at a time, so exepect possibly millions of sequential such small transaction on a billion row table.
> Will their migrations ever run, or will wait for very long periods of time, maybe forever?
Some times, if the load is so very high, migrations will throttle more. At other times, they will push as fast as they can while still keeping to low replication lag threshold. In my experience a gh-ost or vreplication migration is normally good to run even on the busiest times. If a database system is such that it _always_ has substantial replication lag, such that a migration cannot complete in a timely manner, then I'd say the database system is beyond its own capacity anyway, and should be optimized/sharded/whatever.
> How is this possible? Where the migration is running, then? A shadow table, shadow server... none?
So I already mentioned the ghost table. And then, SELECTs are non blocking on the original table.
> What's cut-over?
Cut-over is what we call the final step of the migration: flipping the tables. Specifically, moving away your original table, and renaming the ghost table in its place. This requires a metadata lock, and is the single most critical part of the schema migration, for any tooling involved. This is where something as to give. Tooling such as gh-ost and pt-online-schema-change acquire a metadata lock such that queries are blocked momentarily, until cut-over is complete. With very high load the app will feel it. With extremely high load the database may not be able to (or may not be configured to) accommodate so many blocked queries, and app will see rejections. For low volume load apps may not even notice.
I hope this helps. Obviously this comment cannot accommodate so much more, but hopefully the documentation links I provided are of help.
-
Project mention: Are entity framework tools typically avoided with MySQL & Go and are there alternatives for migration script tooling that version control the entire schema like SSDT? | reddit.com/r/golang | 2021-11-16
Not that I know of. Generally the approach taken is to have ordered migration scripts and a schema version, and then have code automatically apply the appropriate set of migrations in the right order, for example using golang-migrate.
-
-
-
-
Project mention: Any suggestions for a beginner to build a microservice using Go with ES? | reddit.com/r/elasticsearch | 2021-08-11
Has anyone come across any Golang repo like go-clean-arch which uses elasticsearch? As I am a beginner and wanted to have a bit of practice of building microservice using Go with Elasticsearch.
-
Other people maintain https://github.com/volatiletech/sqlboiler which is pretty good, though I wish it were a bit more flexible, but it may be a little faster to get started with.
-
https://github.com/openark/orchestrator might be something for you. I dont know if it works with Mariadb, but it is made for regular old MySQL.
-
I can recommend pairing it with sqlc: https://github.com/kyleconroy/sqlc
-
space-cloud
Open source Firebase + Heroku to develop, scale and secure serverless apps on Kubernetes
-
xo
Command line tool to generate idiomatic Go code for SQL databases supporting PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server (by xo)
I'm a big fan of the database first code generator approach to talking to an SQL database, so much so that I wrote pggen[1] (not to be confused with pggen[2], as far as I can tell a sqlc fork, which I just recently learned about).
I'm a really big partisan of this approach, but I think I'd like to play the devil's advocate here and lay out some of the weaknesses of both a database first approach in general and sqlc in particular.
All database first approaches struggle with SQL metaprogramming when compared with a query builder library or an ORM. For the most part, this isn't an issue. Just writing SQL and using parameters correctly can get you very far, but there are a few times when you really need it. In particular, faceted search and pagination are both most naturally expressed via runtime metaprogramming of the SQL queries that you want to execute.
Another drawback is poor support from the database for this kind of approach. I only really know how postgres does here, and I'm not sure how well other databases expose their queries. When writing one of these tools you have to resort to tricks like creating temporary views in order infer the argument and return types of a query. This is mostly opaque to the user, but results in weird stuff bubbling up to the API like the tool not being able to infer nullability of arguments and return values well and not being able to support stuff like RETURNING in statements. sqlc is pretty brilliant because it works around this by reimplementing the whole parser and type checker for postgres in go, which is awesome, but also a lot of work to maintain and potentially subtlety wrong.
A minor drawback is that you have to retrain your users to write `x = ANY($1)` instead of `x IN ?`. Most ORMs and query builders seem to lean on their metaprogramming abilities to auto-convert array arguments in the host language into tuples. This is terrible and makes it really annoying when you want to actually pass an array into a query with an ORM/query builder, but it's the convention that everyone is used to.
There are some other issues that most of these tools seem to get wrong, but are not impossible in principle to deal with for a database first code generator. The biggest one is correct handling of migrations. Most of these tools, sqlc included, spit out the straight line "obvious" go code that most people would write to scan some data out of a db. They make a struct, then pass each of the field into Scan by reference to get filled in. This works great until you have a query like `SELECT * FROM foos WHERE field = $1` and then run `ALTER TABLE foos ADD COLUMN new_field text`. Now the deployed server is broken and you need to redeploy really fast as soon as you've run migrations. opendoor/pggen handles this, but I'm not aware of other database first code generators that do (though I could definitely have missed one).
Also the article is missing a few more tools in this space. https://github.com/xo/xo. https://github.com/gnormal/gnorm.
-
upper.io/db
Data access layer for PostgreSQL, CockroachDB, MySQL, SQLite and MongoDB with ORM-like features.
upper.io is a viable alternative to GORM. Just a suggestion.
-
octosql
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
-
If you want something agnostic, you can check dbmate that can do this just from CLI.
-
Great, so our data has been backed up! This means that you can restore folks back to a particular point in time when things were as they should be. So long as you have an application that can reliably serve this data (ruminates on articles using Goose to accomplish database versioning) you can restore functionality and value to your users.
-
algernon
:tophat: Small self-contained pure-Go web server with Lua, Markdown, HTTP/2, QUIC, Redis and PostgreSQL support
-
Graphjin a GraphQL to SQL compiler https://github.com/dosco/graphjin
-
-
xorm
xorm是一个简单而强大的Go语言ORM库，通过它可以使数据库操作非常简便。本库是基于原版xorm的定制增强版本，为xorm提供类似ibatis的配置文件及动态SQL支持，支持AcitveRecord操作 (by xormplus)
-
https://github.com/doug-martin/goqu for building SQL queries. Supports MySQL and Postgres at least - super handy!
-
In-database comments combined with something like https://github.com/k1LoW/tbls make for very cheap database documentation.
No affiliation with tbls except that I'm a big fan
Go MySQL related posts
- Transition to FAANG Interviews
- Which SQL driver to use with Postgres
- Boss Says Is Golang losing popularity. True?
- LFX mentorship @ Vitess
- Are entity framework tools typically avoided with MySQL & Go and are there alternatives for migration script tooling that version control the entire schema like SSDT?
- PlanetScale Is Now GA
- Does gorm worth learning?
Index
What are some of the best open-source MySQL projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | tidb | 29,655 |
| 2 | vitess | 12,945 |
| 3 | go-sql-driver/mysql | 11,690 |
| 4 | gh-ost | 9,458 |
| 5 | migrate | 7,585 |
| 6 | usql | 6,845 |
| 7 | kingshard | 5,870 |
| 8 | go-clean-arch | 4,721 |
| 9 | SQLBoiler | 4,440 |
| 10 | orchestrator | 4,317 |
| 11 | sqlc | 4,289 |
| 12 | space-cloud | 3,295 |
| 13 | xo | 2,962 |
| 14 | upper.io/db | 2,812 |
| 15 | octosql | 2,569 |
| 16 | dbmate | 2,182 |
| 17 | goose | 2,012 |
| 18 | algernon | 1,877 |
| 19 | graphjin | 1,599 |
| 20 | gormt | 1,550 |
| 21 | xorm | 1,476 |
| 22 | goqu | 1,325 |
| 23 | tbls | 1,186 |
Are you hiring? Post a new remote job listing for free.
