go-sqlite3 VS sqlc

SQLite encryption at-rest is a hot requested feature of both the “default” CGo driver [1] and the transpiled alternative driver [2]. So, this is a feature I wanted to bring to my own Wasm based Go driver/bindings [3].

Open-source SQLite encryption extensions have had a troubled last few years. For whatever reason, in 2020 the (undocumented) feature that made it easy to offer page-level encryption was removed [4]. Some solutions are stuck with SQLite 3.31.1, but Ulrich Telle stepped up with a VFS approach [5].

Still, their solution seemed harder than something I'd want to maintain, as it requires understanding the structure of what's being written to disk at the VFS layer. So, I looked at full disk encryption for something with less of an impedance mismatch.

Specifically, I'm using the Adiantum tweakable and length-preserving encryption (with 4K blocks, matching the default SQLite page size), and encrypting whole files (rather than page content).

I'm not a cryptographer, so I'd really appreciate some roasting before release.

There is nothing very Go specific about this (apart from the implementation) so if there are no obvious flaws, it may make sense to port it to C/Rust/etc and make it a loadable extension.

[1] https://github.com/mattn/go-sqlite3/pull/1109

for what it's worth, the two pool approach is suggested here by a collaborator to github.com/mattn/go-sqlite3: https://github.com/mattn/go-sqlite3/issues/1179#issuecomment...

SQLite is great. I've also recently settled on it as a key-value store, after considering a few purpose-built key-value solutions. Turns out that it's really easy to make SQLite work as a key-value store, but very difficult to make key-value stores relational.

Just be careful with `:memory:` databases. From the mattn/go-sqlite3 FAQ[1]:

> Each connection to ":memory:" opens a brand new in-memory sql database, so if the stdlib's sql engine happens to open another connection and you've only specified ":memory:", that connection will see a brand new database. A workaround is to use "file::memory:?cache=shared" (or "file:foobar?mode=memory&cache=shared"). Every connection to this string will point to the same in-memory database.

I noticed strange behaviors with just `:memory:` where tables would just disappear at random, and this workaround helped. Make sure to use a unique filename as the `file:` value, especially if using this in tests.

[1]: https://github.com/mattn/go-sqlite3#faq

github.com/mattn/go-sqlite3

For building my web server, I chose to use the Gin framework as the foundation of my app. It was incredibly easy to understand and work with, and I was pleasantly surprised by how seamlessly it integrated with writing unit tests for the server. To handle the database, I leveraged the power of go-sqlite and migrate for efficient SQL queries and migrations. These libraries proved to be both powerful and user-friendly, making the development process a breeze.

https://github.com/mattn/go-sqlite3/blob/master/_example/sim...

SQLite does have an optional "user authentication" extension, though I've not personally tried it out:

https://www.sqlite.org/src/doc/trunk/ext/userauth/user-auth....

The widely used Go SQLite library by mattn says it supports it, if that's useful:

https://github.com/mattn/go-sqlite3#user-authentication

I have an OSS project, sq which is a data-wrangling swiss-army knife for structured data. Think of it as jq for databases. It supports Postgres, SQLServer, MySQL and - relevantly - SQLite. It embeds SQLite via CGo and the mattn/go-sqlite3 driver.

IDE: use whatever make you productive. I personally use vscode. VCS: git, as golang communities use github heavily as base for many libraries. AFAIK Linter: use staticcheck for linting as it looks like mostly used linting tool in go, supported by many also. In Vscode it will be recommended once you install go plugin. Libraries/Framework: actually the standard libraries already included many things you need, decent enough for your day-to-day development cycles(e.g. `net/http`). But here are things for extra: - Struct fields validator: validator - Http server lib: chi router , httprouter , fasthttp (for non standard http implementations, but fast) - Web Framework: echo , gin , fiber , beego , etc - Http client lib: most already covered by stdlib(net/http), so you rarely need extra lib for this, but if you really need some are: resty - CLI: cobra - Config: godotenv , viper - DB Drivers: sqlx , postgre , sqlite , mysql - nosql: redis , mongodb , elasticsearch - ORM: gorm , entgo , sqlc(codegen) - JS Transpiler: gopherjs - GUI: fyne - grpc: grpc - logging: zerolog - test: testify , gomock , dockertest - and many others you can find here

Hi HN, I’m Kyle and together with Andrew (https://news.ycombinator.com/user?id=stanleydrew) we’ve been working on Riza (https://riza.io), a project to make WASM sandboxing more approachable. We’re excited to share a developer preview of our code interpreter API with HN.

There’s a bit of a backstory here. A few months ago, an old coworker reached out asking how to execute untrusted code generated by an LLM. Based on our experience building a plugin system for sqlc (https://sqlc.dev), we thought a sandboxed WASM runtime would be a good fit. A bit of hacking later, we got everything wired up to solve his issue. Now the API is ready for other developers to try out.

The Riza Code Interpreter API is an HTTP interface to various dynamic language interpreters, each running inside a WASM sandbox without access to the outside world (for now). We modeled the API to align with a POSIX shell-style interface.

We made a playground so you can try it out without signing up: https://riza.io

The API documentation lives here: https://docs.riza.io

There are many limitations at the moment, but we expect to rapidly expand capabilities so that programs can e.g. access the network and filesystem. Our roadmap has more details: https://docs.riza.io/reference/roadmap

If you need to execute LLM-generated code we’d love to have you try the API and let us know if you run into any issues. You can email us directly at [email protected].

"Is there a way to get sqlc to use pointers for nullable columns instead of the sql.Null types?"

I came across this yesterday for golang: https://sqlc.dev which is somewhat like what you want, maybe.

Not sure it allows you to parameterize table names but the basic idea is codegen from sql queries so you are working with go code (autocompletion etc).

Agreed, but tools like https://sqlc.dev, which I mention in the article, are a good trade-off that allows you to have verified, testable, SQL in your code.

sqlc is not strictly a conventional ORM. It offers a unique approach by generating Go code from SQL queries. This allows developers to write SQL, which sqlc then converts into type-safe Go code, reducing the boilerplate significantly. It ensures that your queries are syntactically correct and type-safe. sqlc is ideal for those who prefer writing SQL and are looking for an efficient way to integrate it into a Go application.

I was browsing awesome-go for ideas on how to setup my data access layer when I stumbled on sqlc. It seemed like a great option. Code generation is a strategy often used in the Go ecosystem and making my queries safe at compile time was an idea I really liked. Knex was great, but it required of me that I test thoroughly my queries at runtime and that I sanitize my query results to ensure type safety within my application.

Now, we are going to generate the code. For this purpose, we are going to use sqlc.

https://github.com/sqlc-dev/sqlc — for use with //go:generate