github VS litestream

While looking for solutions, I realized that many developers face similar challenges. This issue is widely discussed, particularly in a GitHub thread: Track traffic to GitHub repo longer than 14 days #399.

Probably the older email address is still the primary one for the GitHub account.

GitHub took it upon themselves to change email addresses and author names when merging via the UI buttons like "Squash and Merge" in 2018 and then again in 2019. See <https://github.com/isaacs/github/issues/1368> for the tedious details.

Essentially the post-2019 behaviour seems to be that where possible with "Squash and Merge" they will set noreply@github as the committer so that they can sign the merged commit themselves, and set author name & email to what they have recorded for the GH account involved (and the signature is then a record that GH have verified that account's involvement).

Personally I think it is shocking that they ignore the name and email address that the actual author of the commit has selected. This is both a violation of the author's intentions -- for example, you may set work and personal email addresses in different repositories as discussed here, but GitHub will rewrite them all to the same thing when other people press "Squash and Merge" on your pull requests -- and potentially a doxxing security risk.

I have considered re-reporting this to GitHub via the newer community discussions or via support again, but given the extent to which they've ignored all such reports over the last five years it is hard to find the motivation to do so.

However, there is a massive issue. Github by default truncates insights to t-14 days (where t = today). This is super annoying as there is a discontinuity in data. There is also an archived issue on Github regarding this. The issue has a whopping 119 comments and has been around for over 8 years now. Basically, from the discussions there - Data you don't persist today will be gone 14 days from now. And looks like Github hasn't done anything about it.

> Hi, people have made money using my code and I also don’t care

looks like everyone's missing the point.

> I understand this is upsetting to you

Again, maybe I am on another level of comprehension, so I don't understanda why it is so hard for someone to get it, but I am not upset by that, at all.

I simply know that those who think "it will be fine" are delusional and don't know what they are talking about!

So I just will paste some link to relevant news here, maybe it will make things clearer.

It includes the opinion of Antirez, father of one of the most successful OSS ever: Redis. Maybe his words will open your eyes and tear the veil of Maya.

(spoiler ahead alert!)

Basically you work for free and people don't even thank you and the maintainer ends up being doxed or blamed or pushed aside and in the long term the only solution to keep sanity is to resign

https://www.jeffgeerling.com/blog/2022/burden-open-source-ma...

https://www.theregister.com/2022/01/13/opensource_apacheplc4...

https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-...

https://old.reddit.com/r/linux/comments/z14tt2/reason_why_op...

https://github.com/isaacs/github/issues/167

http://web.archive.org/web/20221217180915/http://antirez.com...

I don't know what the fuss is all about. It was publicly known that Github was breaking automatic git archives consistency for many years. Here is a bug on a project to stop relying on fake github archives (as opposed to stable git-archive(1)):

https://bugzilla.tianocore.org/show_bug.cgi?id=3099

At some point it was impossible to go a few weeks (or even days) without a github archive change (depending on which part of the "CDN" you hit), I guess they must have stabilized it at some point. Here is an old issue before GitHub had a community issue tracker:

https://github.com/isaacs/github/issues/1483

Hello, I see you stepped on my favourite personal soapbox! :)

https://github.com/isaacs/github/issues/1017

I really, really like semi-linear branching/merging. I.e. always rebase-merging, but with a merge commit.

Reasons, in comparison to Github's "rebase merge" which doesn't produce a merge commit:

1. It makes it clear which commits were part of one PR

2. It makes it clear who did the merge

3. It's okay to not have every commit build. but the one being merged will.

4. Still pretty bisectable. You'll narrow things down at least to the PR that caused an issue, and from there it's usually quite simple.

5. Looks very tidy in gitk & Co

Not to say that the feature isn't coming to FOSS git services.. Just that even proprietary organizations have had issues with taking a while to implement them.

Oh this is cool actually! Nice! One of the grievances I have with github commit signing is this issue https://github.com/isaacs/github/issues/1099

In addition, you probably want to read this discussion. https://github.com/isaacs/github/issues/1138

I have not, but I keep meaning to collate everything I've learned into a set of useful defaults just to remind myself what settings I should be enabling and why.

Regarding Litestream, I learned pretty much all I know from their documentation: https://litestream.io/

This presentation is focused on the use-case of vertically scaling a single server and driving everything through that app server, which is running SQLite embedded within your application process.

This is the sweet-spot for SQLite applications, but there have been explorations and advances to running SQLite across a network of app servers. LiteFS (https://fly.io/docs/litefs/), the sibling to Litestream for backups (https://litestream.io), is aimed at precisely this use-case. Similarly, Turso (https://turso.tech) is a new-ish managed database company for running SQLite in a more traditional client-server distribution.

This post intends to help you setup replication for SQLite using Litestream.

I've been messing around with litestream. It is so cool. And, I either found a bug in the -timestamp switch or don't understand it correctly.

What I want to do is time travel into my sqlite database. I'm trying to do some forensics on why my web service returned the wrong data during a production event. Unfortunately, after the event, someone deleted records from the database and I'm unsure what the data looked like and am having trouble recreating the production issue.

Litestream has this great switch: -timestamp. If you use it (AFAICT) you can time travel into your database and go back to the database state at that moment. However, it does not seem to work as I expect it to:

https://github.com/benbjohnson/litestream/issues/564

I have the entirety of the sqlite database from the production event as well. Is there a way I could cycle through the WAL files and restore the database to the point in time before the records I need were deleted?

Will someone take sqlite and compile it into the browser using WASM so I can drag a sqlite database and WAL files into it and then using a timeline slider see all the states of the database over time? :)

We're using SQLite in production very heavily with millions of databases and fairly high operations throughput.

But we did run into some scariness around trying to use Litestream that put me off it for the time being. Litestream is really cool but it is also very much a cool hack and the risk of database corruption issues feels very real.

The scariness I ran into was related to this issue https://github.com/benbjohnson/litestream/issues/510

Litestream is a library that allows you to easily create backups. You can probably just do analytic queries on the backup data and reduce load on your server.

https://litestream.io/

One possible strategy is to have one directory/file per customer which is one SQLite file. But then as the user logs in, you have to look up first what database they should be connected to.

OR somehow derive it from the user ID/username. Keeping all the customer databases in a single directory/disk and then constantly "lite streaming" to S3.

Because each user is isolated, they'll be writing to their own database. But migrations would be a pain. They will have to be rolled out to each database separately.

One upside is, you can give users the ability to take their data with them, any time. It is just a single file.

[0]. https://litestream.io/

Upstream Kuma uses a local SQLite database to store account data, configuration for services to monitor, notification settings, and more. To make sure that our data is available across redeploys, we will bundle Uptime Kuma with Litestream, a project that implements streaming replication for SQLite databases to a remote object storage provider. Effectively, this allows us to treat the local SQLite database as if it were securely stored in a remote database.