drydock
go-sqlite
drydock | go-sqlite | |
---|---|---|
3 | 12 | |
6 | 678 | |
- | - | |
0.0 | 7.5 | |
almost 2 years ago | 2 days ago | |
Go | Go | |
Apache License 2.0 | ISC License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
drydock
-
SQLite in Go, with and Without Cgo
I have been using SQLite in Go projects for a few years now. During early stages of development I always start with SQLite as the main database, then when the project matures, I usually add support for PostgreSQL.
(I usually make a Store interface which is application specific and doesn't even assume there is an SQL database underneath. Then I make "driver" packages for each storage system - be it PostgreSQL, SQLite, flat files, timeseries etc. I have only one set of unit tests that is then run against all drivers. And when I have a caching layer, I also run all the unit tests with or without caching. The cache is usually just an adapter that wraps a Store type. I maintain separate schemas and drivers for each "driver" because I have found that this is actually faster and easier than trying to make generic SQL drivers for instance.)
However, I always keep the SQLite support and it is usually the default when you start up the application without explicitly specifying a database. This means that it is easy for other developers to do ad-hoc experiments or even create integration tests without having to fire up a database, which even when you are able to do it quickly, still takes time and effort. In production you usually want to point to a PostgreSQL (or other) database. Usually, but not always.
I also use it extensively in unit tests (often creating and destroying in-memory databases hundreds of times during just a couple of seconds of tests). I run all my tests on every build while developing and then speed matters a lot. When testing with PostgreSQL I usually set a build tag that specifies that I want to run the tests against PostgreSQL as well. I always want to run all the database tests - I don't always need to run them against PostgreSQL
(Actually, I made a quick hack called Drydock which takes care of creating a PostgreSQL instance and creates one database per test. This is experimental, but I've gotten a lot of use out of it: https://github.com/borud/drydock)
The reason I do this is that it results in much quicker turnaround during the initial phase when the data model may go through several complete rewrites. The lack of friction is significant.
SQLite has actually surprised me. I use it in a project where I routinely have tens of millions of rows in the biggest table. And it still performs well enough at well north of 100M rows. I wouldn't recommend it in production, but for a surprising number of systems you could if you wanted to.
The transpiled SQLite is very interesting to me for two reasons. It makes cross compiling a lot less complex. I make extensive use of Go and SQLite on embedded ARM platforms and then you either have to choose between compiling on the target platform or mess around with C libraries. It also eliminates the need to do two stage Docker builds (which cuts down building Docker images from 50+ seconds to perhaps 4-5 seconds).
The transpiled version is slower by quite a lot. I haven't done a systematic benchmark, but I noticed that a server that stores 30-40 datapoints per second went from 0.5% average CPU load to about 2% average CPU load. I'm not terribly worried about it, but it does mean that when I increase the influx of data I'm most likely going to hit a wall sooner.
I'll be using the transpiled SQLite a lot more in the coming year and I'll be on the Gophers Slack so if anyone is interested in sharing experiences, discussing SQLite in Go, please don't be shy.
-
Exiting the Vietnam of Programming: Our Journey in Dropping the ORM (In Golang)
This isn't new. A lot of applications and libraries do this. And I think it is a good way to design things.
Usually the database I use to develop a SQL schema is Sqlite3, since it allows for really nice testing. Then I add PostgreSQL support (which requires more involved testing setup, but I have a library that makes this somewhat easier: https://github.com/borud/drydock). (SQLite being in C is a bit of a problem since it means I can't get a purely statically linked binary on all platforms - at least I haven't found a way to do that except on Linux. So if anyone has some opinions on alternatives in pure Go, I'm all ears)
In the Java days JDBC every single method implementing some operation would be a lot of boilerplate. JDBC wasn't a very good API. But in Go that is much less of a problem. In part because you have struct tags, and libraries like Sqlx. To that I also add some helper functions to deal with result/error combos. Turns out the majority of my interactions with SQL databases can be carried out in 1-3 lines of code - with a surprising number of cases just being a oneliner. (The performance hit from using Sqlx is in most cases so minimal it doesn't matter. If it matters to you: use Sqlx when modeling and evolving the persistence, and then optimize it out if you must. I think I've done that just once in about 100kLOC worth of code written over the last few years).
And best of all: I get to deal with the database as a database. I write SQL DDL statements to define the schema, and SQL to perform the transactions. I don't have to pretend it is a object model, so I can make full use of the SQL. (Well, actually, I try to make do as far as possible with trivial SQL, but that's a whole different discussion). The interface type takes care of exposing the persistence in a way that fits the application.
(Another thing I've started experimenting with a bit is to return channels or objects containing channels instead of arrays of things. But there is still some experimenting that needs to be done to find a pleasing design)
- Show HN: Idea for unit testing with PostgreSQL in Go
go-sqlite
-
JSON Canvas β An open file format for infinite canvas data
Check out https://github.com/zombiezen/go-sqlite if you're interested in trying out Sqlite in Go again. Nice interface, negligible compile time impact, fast, compiles without CGO. It's very comfortable.
I agree that going from text to sqlite is a bit of a hurdle, especially if you're not writing C :)
-
Jsonfile: A Quick Hack for Tinkering
struggling figuring out how to make my cgo sqlite cross-compile to Windows
Plenty of people trying to fix that.
There's at least:
https://modernc.org/sqlite
Then there's https://github.com/zombiezen/go-sqlite that actually builds https://crawshaw.io/sqlite on top of modernc.
And there's mine that has both a low level and a database/sql driver builds and runs everywhere Go does: https://github.com/ncruces/go-sqlite3
-
Any Full Text Search library for json data?
There are several different Go bindings for SQLite. I maintain https://pkg.go.dev/zombiezen.com/go/sqlite
-
Whatβs your preferred setup to work with SQL DB (without ORM) ?
I like and use https://github.com/zombiezen/go-sqlite for CGo-free SQLite. It avoids some of the problems database/sql has, discussed here: https://crawshaw.io/blog/go-and-sqlite.
- SQLite in Go, with and Without Cgo
- A pure Go embedded SQL database
-
Containerize Go and SQLite with Docker β 9MB Image Size
> C libraries are required to interact with SQLite
Or: modernc.org/sqlite (https://github.com/zombiezen/go-sqlite), "an automatically generated translation of the original C source code of SQLite into Go"
- Gokrazy β A Native Go Userland
-
Library for sqlite3 recommendations?
https://pkg.go.dev/modernc.org/sqlite via https://pkg.go.dev/zombiezen.com/go/sqlite
-
New advanced, CGo-free SQLite package
modernc.org/sqlite provides a database/sql driver, but does not (currently) provide an easy way to get at the more advanced functionality of SQLite, like streaming blob I/O or user-defined functions. David Crawshaw has argued that the database/sql API is not a good fit for SQLite, which is how crawshaw.io/sqlite came about.
What are some alternatives?
tcl
go-sqlite3 - sqlite3 driver for go using database/sql
sqinn - SQLite over stdin/stdout
bbolt - An embedded key/value database for Go.
xgo - Go CGO cross compiler
distroless - π₯ Language focused docker images, minus the operating system.
sqlite - work in progress
bun - SQL-first Golang ORM
framework - PHP Framework providing ActiveRecord models and out of the box CRUD controllers with versioning and ORM support
bun - Incredibly fast JavaScript runtime, bundler, test runner, and package manager β all in one
zeidon-joe - Zeidon Java Object Engine and related projects.
sqlite - Go SQLite3 driver