drydock VS slonik

I have been using SQLite in Go projects for a few years now. During early stages of development I always start with SQLite as the main database, then when the project matures, I usually add support for PostgreSQL.

(I usually make a Store interface which is application specific and doesn't even assume there is an SQL database underneath. Then I make "driver" packages for each storage system - be it PostgreSQL, SQLite, flat files, timeseries etc. I have only one set of unit tests that is then run against all drivers. And when I have a caching layer, I also run all the unit tests with or without caching. The cache is usually just an adapter that wraps a Store type. I maintain separate schemas and drivers for each "driver" because I have found that this is actually faster and easier than trying to make generic SQL drivers for instance.)

However, I always keep the SQLite support and it is usually the default when you start up the application without explicitly specifying a database. This means that it is easy for other developers to do ad-hoc experiments or even create integration tests without having to fire up a database, which even when you are able to do it quickly, still takes time and effort. In production you usually want to point to a PostgreSQL (or other) database. Usually, but not always.

I also use it extensively in unit tests (often creating and destroying in-memory databases hundreds of times during just a couple of seconds of tests). I run all my tests on every build while developing and then speed matters a lot. When testing with PostgreSQL I usually set a build tag that specifies that I want to run the tests against PostgreSQL as well. I always want to run all the database tests - I don't always need to run them against PostgreSQL

(Actually, I made a quick hack called Drydock which takes care of creating a PostgreSQL instance and creates one database per test. This is experimental, but I've gotten a lot of use out of it: https://github.com/borud/drydock)

The reason I do this is that it results in much quicker turnaround during the initial phase when the data model may go through several complete rewrites. The lack of friction is significant.

SQLite has actually surprised me. I use it in a project where I routinely have tens of millions of rows in the biggest table. And it still performs well enough at well north of 100M rows. I wouldn't recommend it in production, but for a surprising number of systems you could if you wanted to.

The transpiled SQLite is very interesting to me for two reasons. It makes cross compiling a lot less complex. I make extensive use of Go and SQLite on embedded ARM platforms and then you either have to choose between compiling on the target platform or mess around with C libraries. It also eliminates the need to do two stage Docker builds (which cuts down building Docker images from 50+ seconds to perhaps 4-5 seconds).

The transpiled version is slower by quite a lot. I haven't done a systematic benchmark, but I noticed that a server that stores 30-40 datapoints per second went from 0.5% average CPU load to about 2% average CPU load. I'm not terribly worried about it, but it does mean that when I increase the influx of data I'm most likely going to hit a wall sooner.

I'll be using the transpiled SQLite a lot more in the coming year and I'll be on the Gophers Slack so if anyone is interested in sharing experiences, discussing SQLite in Go, please don't be shy.

This isn't new. A lot of applications and libraries do this. And I think it is a good way to design things.

Usually the database I use to develop a SQL schema is Sqlite3, since it allows for really nice testing. Then I add PostgreSQL support (which requires more involved testing setup, but I have a library that makes this somewhat easier: https://github.com/borud/drydock). (SQLite being in C is a bit of a problem since it means I can't get a purely statically linked binary on all platforms - at least I haven't found a way to do that except on Linux. So if anyone has some opinions on alternatives in pure Go, I'm all ears)

In the Java days JDBC every single method implementing some operation would be a lot of boilerplate. JDBC wasn't a very good API. But in Go that is much less of a problem. In part because you have struct tags, and libraries like Sqlx. To that I also add some helper functions to deal with result/error combos. Turns out the majority of my interactions with SQL databases can be carried out in 1-3 lines of code - with a surprising number of cases just being a oneliner. (The performance hit from using Sqlx is in most cases so minimal it doesn't matter. If it matters to you: use Sqlx when modeling and evolving the persistence, and then optimize it out if you must. I think I've done that just once in about 100kLOC worth of code written over the last few years).

And best of all: I get to deal with the database as a database. I write SQL DDL statements to define the schema, and SQL to perform the transactions. I don't have to pretend it is a object model, so I can make full use of the SQL. (Well, actually, I try to make do as far as possible with trivial SQL, but that's a whole different discussion). The interface type takes care of exposing the persistence in a way that fits the application.

(Another thing I've started experimenting with a bit is to return channels or objects containing channels instead of arrays of things. But there is still some experimenting that needs to be done to find a pleasing design)

And sometimes invest instead in learning a technology rather than hide it: for example slonik encourages you to write normal SQL queries by making SQL templating easier and safer. In turn, your IDE would be able to understand those queries and give you support based on the database schemas you actually have.

I'd push you to consider using postgres, slonik or similar for database queries. With these libraries, you just write SQL, but they perform input sanitization for you. So you can safely write:

You can already use postgres with Slonik.

https://github.com/gajus/slonik#user-content-slonik-how-are-...

It is not going to be the default because it is way slower.

https://github.com/gajus/slonik/actions/runs/6616647651

Test node_version:18 test_only:postgres-integration is taking 3 minutes.

Test node_version:18 test_only:pg-integration is taking 38 seconds.

For those uninitiated, Slonik is a battle-tested SQL query building and execution library for Node.js. Its primary goal is to allow you to write and compose SQL queries in a safe and convenient way. Now, let's see how it pairs with Express.js.

I am the maintainer of Slonik and I am trying to understand what portion of this sub-users are using Slonik vs other libraries, and if they are using anything else – what are their reasons for it.

It's nice that they implemented string templates essentially exactly the same way Javascript template literals and tag functions work. They even give an example of using it to create a prepared statement (e.g. DB."SELECT * FROM foo WHERE bar = \{inputParam}") which is exactly what many NodeJS libraries due, e.g. Slonik https://github.com/gajus/slonik, like sql`SELECT * FROM foo WHERE bar = ${inputParam}`;

I've found libraries like Zod useful when interacting with external data sources like a database. Slonik[1] uses Zod to define the types expected from a SQL query and then performs runtime validation on the data to ensure that the query is yielding the expected type.

I don't think it's necessary to use Zod/runtime validation everywhere, but it's a nice tool to have on hand.

[1]https://github.com/gajus/slonik

Demonstrate how easily and accidentally one can make an SQL injection with these:

https://github.com/porsager/postgres

https://github.com/gajus/slonik

Have you tried Slonik (https://github.com/gajus/slonik)? It won't generate types from queries automatically, but it encourages writing SQL vs. a query builder and allows type annotations of queries with Zod. Query results are validated at runtime to ensure the queries are typed correctly.