github-to-sqlite VS til

Yeah, I worry about that too. I have SO much of my stuff dependent on my GitHub account now.

I'm slightly reassured by how useful their APIs are. I have automated exports of a lot of my GitHub issues, though I really should shore those up and make sure I'm capturing everything.

That's one of the reasons I built https://github.com/dogsheep/github-to-sqlite

I have trouble answering this question myself, and I created it!

The problem I have is that it can be applied to too many different problems.

I personally have used it for the following (a truncated summary):

- Publishing data online to allow other people to explore it, for example https://scotrail.datasette.io and https://russian-ira-facebook-ads.datasettes.com/

- Building websites, by combining it with custom templates. https://datasette.io and https://www.niche-museums.com and https://til.simonwillison.net are three examples

- Building my own combined search engine over a bunch of different data. https://github-to-sqlite.dogsheep.net is this for my GitHub issues and commits and issue comments across 100+ projects

- Similarly, building a code search engine across multiple repos (partly to demonstrate how far you can go with custom plugins): https://ripgrep.datasette.io

- Any time I have a CSV file I open it in the Datasette Desktop macOS app first to start exploring it: https://datasette.io/desktop

- As a prototyping tool. It's the fastest way I know of to get from some data files (CSV or JSON) to a working JSON API - and a GraphQL API too using this plugin: https://datasette.io/plugins/datasette-graphql

- Messing around with geospatial data - here's a write-up of my favourite experiment with that so far: https://simonwillison.net/2021/Jan/24/drawing-shapes-spatial...

This is a bewilderingly wide array of things! And I keep on finding new problems I can apply it to:

Of course, if all you have is a hammer, everything looks like a nail. But thanks to the plugin system (and the amazing flexibility of SQLite under the good) I can reshape my hammer into all sorts of interesting shapes!

I've been trying to capture some of this at https://datasette.io/for

This is one of my biggest marketing challenges for the project though. If someone asks you for an elevator pitch you need to do better than spending 15 minutes talking through a wide ranging bulleted list!

You might also want to look at https://github.com/dogsheep/github-to-sqlite for saving issues, pull-requests, etc

Thanks for letting me know - fixed that here: https://github.com/dogsheep/github-to-sqlite/issues/74

My application Datasette can serve this need: https://datasette.io/

Lots of live demos - two of my favourites are https://github-to-sqlite.dogsheep.net/ and https://global-power-plants.datasettes.com/

I've been using the sqlite_utils Python library to build a whole suite of tools for importing data from various different sources into SQLite, including healthkit-to-sqlite, swarm-to-sqlite, twitter-to-sqlite, inaturalist-to-sqlite, google-takeout-to-sqlite, github-to-sqlite, genome-to-sqlite, pocket-to-sqlite, hacker-news-to-sqlite, evernote-to-sqlite.

> If you learn something the hard way, share your findings with others. You have blazed a new trail; now you must mark it for your fellow travellers. Sharing knowledge is an unreasonably effective way of helping others.

This is a really nice philosophy. It's one of the reasons why I have my https://til.simonwillison.net TIL site - any time I search for something and can't find the answer is a hint that there's a tiny gap in the internet which I can help fill.

Hosting these on GitHub is such a good idea:

- GitHub have world class backups - commit something there and it gets replicated to data centers on three continents (I believe) - and a free public repo there won't vanish if you forget to update an expired credit card

- Related: GitHub is free. I care about this not because of not wanting to pay now, but because I don't want my content to be at risk if I forget to pay in the future (or can't pay for whatever reason)

- GitHub has several great web UIs for editing content, in addition to being able to edit in any other tool that supports the git protocol

- GitHub Actions makes it possible to add all sorts of automations on top of your notes, again for free. I use that to deploy my custom https://til.simonwillison.net site (mainly to give myself search)

- GitHub's own search is pretty good though!

- You can also use GitHub Pages if you just want a custom static site version of your notes.

- if someone spots a typo in your notes they can submit a PR to fix it!

That's awesome. Parts of that sound a little bit like how my https://til.simonwillison.net/ site works.

I started publishing "TIL" posts a few years ago and everything in this post here resonated 100% with my experience of writing those.

The great thing about TILs is that once you form a solid set of habits around them they can be extremely quick to put together: the majority of my TIL posts take between 15 minutes and half an hour to write.

I make extensive personal notes on everything I'm doing (in GitHub issues threads or VS Code scratch documents) - turning those into a TIL is mainly about pasting those notes into a Markdown file and tidying them up a bit.

https://til.simonwillison.net/ is my collection so far.

I get a huge amount of value out of these. I don't particularly care if other people read them, the value is in helping me better understand the material and enabling me to refer back to them in the future.

I refer to some of them multiple times every week! This one for example: https://til.simonwillison.net/python/pyproject

I've started to consider "commit to writing about it" as the price I have to pay for giving into the lure of another project. It's one of the main reasons I publish so much content on https://simonwillison.net/ and https://til.simonwillison.net

A project with a published write-up unlocks so much more value than one which you complete without giving others a chance of understanding what you built.

I've maintained internal blogs (sometimes just a Slack channel or Confluence area) at previous employers for this purpose too.

https://github.com/simonw/til/blob/main/llms/llama-7b-m2.md

I'm definitely keen on suggestions for improvements I can make to the default UI.

Datasette provides both a JSON API (easily enabled for CORS access) and supports custom templates, so it's possible to customize the UI any way you like.

So far I've not seen many examples of extensive customization. I use the custom templates a lot myself - these four sites are all just regular Datasette with some custom templates:

- https://datasette.io/

- https://til.simonwillison.net/

- https://www.niche-museums.com/

- https://www.rockybeaches.com/us/pillar-point

Source code is on GitHub for all four.

I have trouble answering this question myself, and I created it!

The problem I have is that it can be applied to too many different problems.

I personally have used it for the following (a truncated summary):

- Publishing data online to allow other people to explore it, for example https://scotrail.datasette.io and https://russian-ira-facebook-ads.datasettes.com/

- Building websites, by combining it with custom templates. https://datasette.io and https://www.niche-museums.com and https://til.simonwillison.net are three examples

- Building my own combined search engine over a bunch of different data. https://github-to-sqlite.dogsheep.net is this for my GitHub issues and commits and issue comments across 100+ projects

- Similarly, building a code search engine across multiple repos (partly to demonstrate how far you can go with custom plugins): https://ripgrep.datasette.io

- Any time I have a CSV file I open it in the Datasette Desktop macOS app first to start exploring it: https://datasette.io/desktop

- As a prototyping tool. It's the fastest way I know of to get from some data files (CSV or JSON) to a working JSON API - and a GraphQL API too using this plugin: https://datasette.io/plugins/datasette-graphql

- Messing around with geospatial data - here's a write-up of my favourite experiment with that so far: https://simonwillison.net/2021/Jan/24/drawing-shapes-spatial...

This is a bewilderingly wide array of things! And I keep on finding new problems I can apply it to:

Of course, if all you have is a hammer, everything looks like a nail. But thanks to the plugin system (and the amazing flexibility of SQLite under the good) I can reshape my hammer into all sorts of interesting shapes!

I've been trying to capture some of this at https://datasette.io/for

This is one of my biggest marketing challenges for the project though. If someone asks you for an elevator pitch you need to do better than spending 15 minutes talking through a wide ranging bulleted list!

Make notes, in private and in public.

I have a private "notes" repo on GitHub where I keep notes in the issues (the repo itself is empty). Any time I'm trying out a new piece of software I open an issue there, then add notes on the issue comments as I figure things out.

I use GitHub issues because they have excellent backups and they show up on GitHub search - plus there's a really good API which I use to periodically export and backup my note elsewhere.

If something fits. I'll turn my notes into a TIL and publish them on https://til.simonwillison.net - that site uses the markdown format as GitHub issues, so publishing a TIL that started out as an issue comment only takes me a few minutes.