datasette.io VS TiddlyWiki

Opened an issue exploring alternatives here: https://github.com/simonw/datasette.io/issues/109

I decided to just drop "any size" but keep "any shape".

Here's an example from one of my projects: https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml

I've come around to almost the opposite approach.

I pull all of the data I can get my hands on (from Twitter, GitHub, Swarm, Apple Health, Pocket, Apple Photos and more) into SQLite database tables that match the schema of the system that they are imported from.

For my own personal Dogsheep (https://simonwillison.net/2020/Nov/14/personal-data-warehous...) that's 119 tables right now.

Then I use SQL queries against those tables to extract and combine data in ways that are useful to me.

If the schema of the systems I am importing from changes, I can update my queries to compensate for the change.

This protects me from having to solve for a standard schema up front - I take whatever those systems give me. But it lets me combine and search across all of the data from disparate systems essentially at runtime.

I even have a search engine for this, which is populated by SQL queries against the different source tables. You can see an example of how that works at https://github.com/simonw/datasette.io/blob/main/templates/d... - which powers the search interface at https://datasette.io/-/beta

I've been exploring this technique more over the past year and I really like it - https://datasette.io (code at https://github.com/simonw/datasette.io ) is a more recent and much more complicated example.

Extracting links from markdown and using them to populate some additional columns or tables at build time would be pretty straight forward.

I run SQLite in serverless environments (Cloud Run, Vercel, Heroku) for dozens of projects... but the trick is that they all treat the database as a read-only asset.

If I want to deploy updated data, I build a brand new image and deploy the application bundled with the data. I tend to run the deploys for these (including the database build) in GitHub Actions workflows.

This works really well, but only for applications that don't need to apply constant updates more than a few times an hour! If you have a constant stream of updates I still think you're better off using a hosted database like Heroku PostgreSQL or Google Cloud SQL.

One example of a site I deploy like that is https://datasette.io/ - it's built and deployed by this GitHub Actions workflow here: https://github.com/simonw/datasette.io/blob/main/.github/wor...

> What does ownership mean here?

It means owning the code and the data. With webapps, the code and data are hosted and owned, the users do not own the code, cannot run it independently. This is a clear dileneation between owner and user, and the owners can use that clear line to create artificial scarcity of various kinds. (The most popular being the subscription SaaS model). It's also easier to defend your IP since end users never see your binaries.

I like to make my software single html files whenever possible. People can just save them and run them locally. Havent met anyone who cares yet though.

I like that idea a lot, and I care. I think others care, but yes, it's a niche interest. Take a look at https://tiddlywiki.com/ for an example of a fairly successful project that uses the single html format running locally. However it suffers from limitations on File|Save which often requires a separate runtime of some kind to support.

Another project that approaches this ideal is https://redbean.dev/, @jart's tiny, performant, featureful single-file webserver. In this case the "single file" is a server executable + zip whose state must be updated on the command-line, but I think hits a sweet spot in terms of practicality, and a global minima when it comes to minimizing dependencies. (Redbean bundles SQLite and Lua so it's also possible to do through-the-web state updates as in a traditional webapp.)

My own project, Simpatico, aspires to be something along these lines. Eventually your browser tab is both a client and server process, connecting via websockets to other connected browsers, storing all state locally. I call this pattern "monomorphism", a play on the "isomorphic" javascript SPA. The server[2] is currently written in ~1 node file, but eventually I would like to port to redbean (and greenbean, the websocket version of redbean, but it isn't quite ready yet). The server grew several features to support a fast, practical BTD loop using markdown[1], and safe, performant execution on the public internet[2], but ultimately I'd like to pare it down to serving a single html file and allow the connected clients to provide all diversity of experience. I've used it to explore all kinds of browser apis, from crypto[3] to svg[4] to writing my own libraries (combine[4] and stree[5]). And it's all running locally, and easily hosted on a $5 VPS, and its all open source.

1 - https://simpatico.io/lit.md

2 - https://simpatico.io/reflector

3 - https://simpatico.io/crypto

4 - https://simpatico.io/combine

5 - https://simpatico.io/stree

I use TiddlyWiki. It's a portable editable wiki that doesn't require a web server or web hosting. You open it from your computer, edit it, and save it. You get all of the linking that you'd expect to see in a wiki, and it's super readable and easy to use.

It is a single-HTML-file TiddlyWiki instance that runs in a web browser (offline as well as online), meant to be downloaded and stored wherever suits you best. Everything that you see when working in BASIC Anywhere Machine (everything that makes "BAM" work as an IDE and all BASIC programs) exist in the one HTML file.

TiddlyWiki still works as intended: https://tiddlywiki.com/#GettingStarted but there are so many different clients to run on. Mobile or Desktop ? What OS? What Browser?

This effort https://val.packett.cool/blog/tiddlypwa/ is remarkable as the mobile side of saving is not as robust as on the desktop side of things and there is a scaling limit on performance as the number of tiddlers grows. Also the syncing between tw documents between different desktop/mobile clients can be a challenge with diffing.

Since then I've moved back to plain vanilla vim for a wiki (map gf :tabe ) but tw.html is still good for data other than plain text and TiddlyPWA https://tiddly.packett.cool/ is a great effort to revisit TiddlyWiki again.

You should check out TiddlyWiki as it’s designed around the concept that small linkable notes are the best way to organize.

https://tiddlywiki.com/

It’s html based so you can access it in the same way you would access a website but it can be locally stored. Saving is a bit tricky but there are multiple solutions detailed on their site. https://tiddlywiki.com/

Hi, you can use tw on nodejs with npm package tiddlywiki....