timeliner VS datasette

Heya! I'm the author of PhotoStructure, and my Google Photos account (before I started working on PhotoStructure) is about that size, too.

I wrote up some tips here: https://photostructure.com/faq/takeout/

This is what I did:

1. First try to fetch all your Google Photos via Takeout in one archive. If it fails (like it did for me), try different-sized .tgz archives. I had to use the 10 Gb option (using 50gb caused an internal-to-google error).

If that fails to work, the last resort is to manually create by-year albums, shove all photos from that year into that album, and do a takeout of just that album. Repeat as necessary for every year.

2. Install an app on your phone to *directly* upload the original photos and videos from your phone to your NAS/home server. I have several recommended apps here: https://photostructure.com/faq/how-do-i-safely-store-files/#...

At this point, you can still use Google Photos (for viewing and as a last-ditch backup), but your originals are safe (without all the Google Photo downsampling and metadata shenanigans), and you're free to use whatever self-hosted software you want (like PhotoStructure, but there are a ton of alternatives, as well).

FWIW, I also tried this software: https://github.com/mholt/timeliner -- it does what it can, but the files you get via the API has a bunch of metadata stripped from it. I even had captured-at times get mangled with older photos.

This is why I'm building Timelinize [1]. It's a follow-up to my open source Timeliner project [2], which has the potential to download all your digital life onto your own computer locally, and projects it all onto a single timeline, across all data sources (text messages, social media sites, photos, location history, and more).

It's a little different from "self hosting" but it does have a similar effect of bringing all your data home and putting it in your control.

The backend and underlying processing engine is all functional and working very well; now I'm just getting the UI put together, so I hope to have something to share later this year.

[1]: https://twitter.com/timelinize (website coming eventually)

[2]: https://github.com/mholt/timeliner

Not a "big project/service" but a Go project that uses Sqlite is one of my own, Timeliner[1] and its successor, Timelinize[2] (still in development). Yeah the cgo dependency kinda sucks but you don't feel it in code, just compilation. And it easily manages Timeline databases of a million and more entries just fine.

[1]: https://github.com/mholt/timeliner

[2]: https://twitter.com/timelinize

This looks promising but might be a bit complicated for you: https://github.com/mholt/timeliner

I made Timeliner to download my Google Photos: https://github.com/mholt/timeliner -- requires some tech prowess for now, though.

Seems like a perfect job for Datasette: SQLite plus web api and UI

“Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API.”

https://datasette.io/

You might like this: https://datasette.io/ and it might even fill your blogging needs. Fully opensource (and there are extension to use llm's with it).

I wrote an async wrapper around SQLite in Python - I'm using a thread pool: https://github.com/simonw/datasette/blob/main/datasette/data...

I have multiple threads for reads and a single dedicated thread for writes, which I send operations to via a queue. That way I avoid ever having two writes against the same connection at the same time.

I've been working on this for almost six years now. The initial idea was to solve the "get stuff into a SQLite database" problem as effectively as possible, because my https://datasette.io/ project was only useful if you first get your data into SQLite.

It's since grown to handle all manner of manipulations. Possibly the most useful is its support for advanced schema alterations via the "transform" CLI command (and accompanying table.transform() Python method):

https://sqlite-utils.datasette.io/en/stable/cli.html#transfo...

https://sqlite-utils.datasette.io/en/stable/python-api.html#...

This addresses one of the most frequent complaints people have about SQLite - that it doesn't support a range of table alter operations beyond simple things like adding a new column.

sqlite-utils transform is an implementation of the pattern described in the SQLite docs - https://www.sqlite.org/lang_altertable.html#otheralter - where you create a new empty table with the modified schema, then copy the old data across and rename the tables as part of a single transaction.

Simon Willison's github would be a great place to get started imo -

https://github.com/simonw/datasette

I'm a fan on simonw's datasette/dogsheep ecosystem https://datasette.io/

I use Anki the exact same way. After a lifetime of learning I have accepted that I will never read over anything I write for myself voluntarily - so my two options are:

1. Write an article so good I can publish it and look it over myself later on. I did this last year with https://andrew-quinn.me/fzf/, for example.

2. Create Anki cards out of the material. Use the builtin Card Browser or even https://datasette.io/ on the underlying SQLite database in a pinch to search for my notes any time I have to.

Were you aware of, or tempted by https://datasette.io/ for creating your solution?