google-takeout-to-sqlite VS github-to-sqlite

I've been using the sqlite_utils Python library to build a whole suite of tools for importing data from various different sources into SQLite, including healthkit-to-sqlite, swarm-to-sqlite, twitter-to-sqlite, inaturalist-to-sqlite, google-takeout-to-sqlite, github-to-sqlite, genome-to-sqlite, pocket-to-sqlite, hacker-news-to-sqlite, evernote-to-sqlite.

Yeah, I worry about that too. I have SO much of my stuff dependent on my GitHub account now.

I'm slightly reassured by how useful their APIs are. I have automated exports of a lot of my GitHub issues, though I really should shore those up and make sure I'm capturing everything.

That's one of the reasons I built https://github.com/dogsheep/github-to-sqlite

I have trouble answering this question myself, and I created it!

The problem I have is that it can be applied to too many different problems.

I personally have used it for the following (a truncated summary):

- Publishing data online to allow other people to explore it, for example https://scotrail.datasette.io and https://russian-ira-facebook-ads.datasettes.com/

- Building websites, by combining it with custom templates. https://datasette.io and https://www.niche-museums.com and https://til.simonwillison.net are three examples

- Building my own combined search engine over a bunch of different data. https://github-to-sqlite.dogsheep.net is this for my GitHub issues and commits and issue comments across 100+ projects

- Similarly, building a code search engine across multiple repos (partly to demonstrate how far you can go with custom plugins): https://ripgrep.datasette.io

- Any time I have a CSV file I open it in the Datasette Desktop macOS app first to start exploring it: https://datasette.io/desktop

- As a prototyping tool. It's the fastest way I know of to get from some data files (CSV or JSON) to a working JSON API - and a GraphQL API too using this plugin: https://datasette.io/plugins/datasette-graphql

- Messing around with geospatial data - here's a write-up of my favourite experiment with that so far: https://simonwillison.net/2021/Jan/24/drawing-shapes-spatial...

This is a bewilderingly wide array of things! And I keep on finding new problems I can apply it to:

Of course, if all you have is a hammer, everything looks like a nail. But thanks to the plugin system (and the amazing flexibility of SQLite under the good) I can reshape my hammer into all sorts of interesting shapes!

I've been trying to capture some of this at https://datasette.io/for

This is one of my biggest marketing challenges for the project though. If someone asks you for an elevator pitch you need to do better than spending 15 minutes talking through a wide ranging bulleted list!

You might also want to look at https://github.com/dogsheep/github-to-sqlite for saving issues, pull-requests, etc

Thanks for letting me know - fixed that here: https://github.com/dogsheep/github-to-sqlite/issues/74

My application Datasette can serve this need: https://datasette.io/

Lots of live demos - two of my favourites are https://github-to-sqlite.dogsheep.net/ and https://global-power-plants.datasettes.com/

I've been using the sqlite_utils Python library to build a whole suite of tools for importing data from various different sources into SQLite, including healthkit-to-sqlite, swarm-to-sqlite, twitter-to-sqlite, inaturalist-to-sqlite, google-takeout-to-sqlite, github-to-sqlite, genome-to-sqlite, pocket-to-sqlite, hacker-news-to-sqlite, evernote-to-sqlite.