datasette.io VS openapi-generator

Opened an issue exploring alternatives here: https://github.com/simonw/datasette.io/issues/109

I decided to just drop "any size" but keep "any shape".

Here's an example from one of my projects: https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml

I've come around to almost the opposite approach.

I pull all of the data I can get my hands on (from Twitter, GitHub, Swarm, Apple Health, Pocket, Apple Photos and more) into SQLite database tables that match the schema of the system that they are imported from.

For my own personal Dogsheep (https://simonwillison.net/2020/Nov/14/personal-data-warehous...) that's 119 tables right now.

Then I use SQL queries against those tables to extract and combine data in ways that are useful to me.

If the schema of the systems I am importing from changes, I can update my queries to compensate for the change.

This protects me from having to solve for a standard schema up front - I take whatever those systems give me. But it lets me combine and search across all of the data from disparate systems essentially at runtime.

I even have a search engine for this, which is populated by SQL queries against the different source tables. You can see an example of how that works at https://github.com/simonw/datasette.io/blob/main/templates/d... - which powers the search interface at https://datasette.io/-/beta

I've been exploring this technique more over the past year and I really like it - https://datasette.io (code at https://github.com/simonw/datasette.io ) is a more recent and much more complicated example.

Extracting links from markdown and using them to populate some additional columns or tables at build time would be pretty straight forward.

I run SQLite in serverless environments (Cloud Run, Vercel, Heroku) for dozens of projects... but the trick is that they all treat the database as a read-only asset.

If I want to deploy updated data, I build a brand new image and deploy the application bundled with the data. I tend to run the deploys for these (including the database build) in GitHub Actions workflows.

This works really well, but only for applications that don't need to apply constant updates more than a few times an hour! If you have a constant stream of updates I still think you're better off using a hosted database like Heroku PostgreSQL or Google Cloud SQL.

One example of a site I deploy like that is https://datasette.io/ - it's built and deployed by this GitHub Actions workflow here: https://github.com/simonw/datasette.io/blob/main/.github/wor...

Disclaimer: We're an early adopter of Stainless at Mux.

I've spent more of my time than I'd like to admit managing both OpenAPi spec files [1] and fighting with openapi-generator [2] than any sane person should have to. While it's great having the freedom to change the templates an thus generated SDKs you get with using that sort of approach, it's also super time consuming, and when you have a lot of SDKs (we have 6 generated SDKs), in my experience it needs someone devoted to managing the process, staying up with template changes etc.

Excited to see more SDK languages come to Stainless!

[1] https://www.mux.com/blog/an-adventure-in-openapi-v3-api-code...

[2] https://github.com/OpenAPITools/openapi-generator

As a result, the following specification can be used to generate clients in a number of different languages via OpenAPI Generator.

Of course you can compile the server from source if you have Go and the OpenAPI generator JAR (https://github.com/OpenAPITools/openapi-generator?tab=readme...)

Follow these steps : https://github.com/c100k/rebootx-on-prem/blob/master/.github...

And then :

(cd ./impl/http-server-go && GOARCH=amd64 GOOS=openbsd go build -o /app/rebootx-on-prem-http-server-go-openbsd-amd64 -v)

By adapting the arch if needed. Not tested, but it should work.

TL;DR: Start generating your HTTP clients and all the DTOs of the requests and responses automatically from your API, using openapi-generator instead of writing your own.

As an alternative, you can also use the official OpenAPI Generator, which is a more generic tool supporting a wide range of languages and frameworks.

I trialed generating SDKs using the OpenAPI Generator package, which was largely unsatisfactory.

If Swagger/OpenAPI is available, save yourself a lot of trouble and generate the client using OpenAPI Generator. If not, use a library like RestEase to make it significantly easier to create the client.

For a run of the mill REST API you should generate OpenAPI (Swagger) info for the API using a library like NSwag or Swashbuckle. You'd want to do this no matter what because it's documentation for the API, but the bonus is that you can use it with tools like OpenAPI Generator to create API client code and models in a variety of languages. You certainly can create an API client library manually, it would entail having a nuget package with a class library that contains the models and client code for calling the endpoints (which I'd create using a lib such as RestEase unless you just enjoy writing boilerplate code by hand). However 95% of the time it simply isn't worth creating your own lib when OpenAPI is available because once you've done it a time or two it takes less than 5 min to run the generator and create (or update) a lib.

Then you can use oapi-codegen or openapi-generator to generate the Go (or other language) SDK for it.