SocketCluster VS litestream

I've been advocating for this approach for years. You can't avoid race conditions without using some kind of queueing mechanism and you need backpressure to measure queue capacity. I built this into every aspect of https://socketcluster.io/ - From pub/sub channels, RPCs to event listeners.

It's interesting reading this.

Preventing these kinds of concurrency issues is exactly why I built https://socketcluster.io years ago. Though it solves the problem at the app layer rather than the storage layer.

But not many developers care about these race conditions it seems.

It's not just an issue with SQL but a more general issue with coding.

This is a great example because it shows how concurrent executions can lead to significant issues.

My project https://github.com/socketCluster/socketcluster has been accumulating stars slowly but steadily over 15 years. Now it has over 6k stars but it doesn't seem to mean much nowadays as a metric. It sucks having put in the effort and seeing it get lost in a sea of scams.

Well I built it up in parts and brought the parts together over time. SocketCluster has over 6k stars on GitHub. https://github.com/socketCluster/socketcluster

Saasufy itself isn't open source but I'm planning to sell licenses of the code. SocketCluster is a core component of Saasufy. The goal did evolve slightly; originally, it was to make it easier to build full stack applications. Now it's actually; lets you build entire full stack apps without code. That bigger goal has been achieved. I have some videos linked from the Docs page showing how it works.

https://socketcluster.io/ has had such stream implementation and backpressure management since at least 2019;.

https://github.com/SocketCluster/writable-consumable-stream

The problem with HTTP2 is that the server-push aspect was tacked on top of an existing protocol as an afterthought. Also, because HTTP is a resource transfer protocol, it adds a whole bunch of overheads like request and response headings which aren't always necessary but add to processing time. The primary purpose of HTTP2 was to allow servers to preemptively push files/resources to clients to avoid round-trip latency; to reduce the reliance on script bundles.

WebSockets is a simpler protocol built from the ground up for bidirectional communication. It provides a lot more control over the flow of data as everything passes over a single connection which has a single lifecycle. It makes it a lot easier to manage state and to recover cleanly from a lost connection when you only have one logical connection. It makes it easier to process messages in a specific order and to do serial processing of messages.

I considered the possibility of switching the transport to HTTP2 for https://socketcluster.io/ years ago, but it's a fundamentally more complex protocol which adds unnecessary overheads and introduces new security challenges so it wasn't worth it.

For WebSockets, using SocketCluster (https://socketcluster.io), it's possible to queue up all requests from the same client and then detect and respond to high backpressure spikes (e.g. by disconnecting the client and/or recording the incident).

You can combine different approaches like limiting the number of connections from a single IP within a certain timeframe and also limiting the backpressure.

The ability to measure and respond to extreme backpressure spikes on a per-end-user basis is highly valuable because backpressure in SocketCluster takes into account the processing time of client requests.

A common strategy that spammers use is to identify and invoke the most expensive endpoints in your system.

A lot of people still don't understand the value proposition of being able to process requests from clients in-order. It's also really good at preventing race conditions and makes your environment highly predictable.

I never fully understood the need for back end message queues TBH. You can just poll the database or data store every few seconds and process tasks in batches... IMO, the 'real time' aspect was only ever useful for front end use cases for performance reasons since short polling every second with HTTP (with all its headers/overheads) is prohibitively expensive. Also, HTTP long polling introduces some architectural complexity which is not worth it (e.g. sticky sessions are required when you have multiple app servers).

Unfortunately, real-time messaging complexity entirely to the back end has been the norm for a very long time. My experience is that, in general, it makes the architecture way more difficult to manage. I've been promoting end-to-end pub/sub as an alternative for over a decade (see https://socketcluster.io/) but it mostly fell on deaf ears. Most devs just don't realize how much complexity is added by micromanaging pub/sub channels on the back end and figuring out which message belongs to what user instead of letting end-users decide what channels to subscribe to directly.

I think part of the problem was the separation between front end and back end developer responsibilities. Back end developers like to ignore the front end as much as possible; when it comes to architecture, their thinking rarely extends beyond the API endpoint; gains which can be made from better integrating the back end with the front end is 'not their job'. From the perspective of front-end developers, they see anything performance-related as 'not their job' either... There aren't enough full stack developers with the required insights to push for integration efficiency/simplicity.

Recently, I added an AI-generated soundtrack to my open source project's home page https://socketcluster.io/

It seems unconventional at first but I distinctly remember about a decade ago when Adobe Flash was still broadly supported, many Flash websites had soundtracks. I think the reason why regular HTML websites didn't have them was because it was difficult to implement and internet was much slower so they had to be streamed in a special way.

This is one of a bunch of issues that have been popping up since a vibe coder took over the bulk of development on this project. There's a (probably also AI generated) list of a big portion of the issues here: https://github.com/benbjohnson/litestream/issues/1221. That proposal has been open for a few months, and it seems (from my POV) unlikely to be resolved any time soon.

It’s not, but you could do something like https://litestream.io/ and just continuously replicate it to pretend to be multi-user

would https://litestream.io/ be a good solution here?

Consider SQLite replication for backups using Litestream.

My guess would be that performance improvements (mostly hardware from Moore's law and the proliferation of SSDs, but also SQLite itself) have led to far fewer websites needing to run on more than 1 computer, and most are fine on a $5/month VPS

And stuff like https://litestream.io/ or SQLite adding STRICT mode

The Solution: Litestream.

Litestream + SQLite: https://litestream.io

* https://litestream.io/ Replication to S3 (or compatible) - more disaster recovery than fail over

Would https://litestream.io/ perhaps help?

>Thx. The only problem I have with litestream binary is ~31Mb !?

Where are you seeing a 31 MB binary? Latest releases are 10-11 MB.[0]

>This why I prefer to take backup stuff in a side container, eg: https://github.com/atrakic/gin-sqlite/blob/main/compose.yml

Yeah, I agree that's cleaner, but once you're dealing with multiple containers, it's a big step up in complexity, and you can't do the simple install on places like Fly.io and Lightsail.

>As a side note, you might consider revisiting your dockerfiles and skip litestream build steps, eg. in your final stage just add line like this:

>COPY --from=litestream/litestream /usr/local/bin/litestream /bin/litestream

Ah, thanks! The litestream Docker image is new as of 0.5.0.

[0] https://github.com/benbjohnson/litestream/releases