kine VS harbormaster

-- Kine (Kine is not etcd)

Looking at [1], is there any reason why this couldn’t run in something like DynamoDB? I feel like the operations required would fit

https://github.com/k3s-io/kine/blob/master/pkg/drivers/gener...

I don't like some of the decisions that k3s makes which is why I stay away from it, but it's a great project. I'm not using kine yet, but I fantasize about using it exclusively every time I deal with etcd-induced slowness/random CPU spikes on my group of controller nodes. k0s supports kine too of course but I just haven't switched it on yet.

Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0].

> I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.

As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer).

One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO.

If you do build a plugin system though, you can pass off the work of maintaining the other implementations!

[0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all

[1]: https://github.com/k3s-io/kine

I could spent >0s looking into this, but I've been curious how kline implements Etcd's Watch API. For the matter, I don't have much idea in general how kline handles data. Kline would be a great candidate for a blog post or introductory technical article on.

I do quickly see one issue- evidently watches presently use polling. The issue requests use of postgres triggers: https://github.com/k3s-io/kine/issues/20

check out https://github.com/k3s-io/kine

Not to be mistaken for high-availability Dqlite[1], which is one of the options one can run the k3s kubernetes distribution on (instead of etcd), via the Kine etcd shim[2]. Ultimately though the K3s team replaced Dqlite with an embedded etcd to get high-availability[3].

[1] https://dqlite.io/

[2] https://github.com/k3s-io/kine

[3] https://rancher.com/docs/k3s/latest/en/installation/ha-embed...

I use an HP ProLiant Microserver with four drives in a ZFS RAIDZ array and an SSD for the OS. For software, I mostly run it in Docker using a very small container orchestration program I wrote:

https://gitlab.com/stavros/harbormaster

I needed something that would restart containers automatically when I pushed to a branch, so I wrote a few lines of code to do it:

https://gitlab.com/stavros/harbormaster

As far as PaaSes go, it's probably the simplest, and works really well.

I was in the same boat as you and built something simple that I really like:

https://gitlab.com/stavros/harbormaster

It'll just pull some repos, make sure the containers are up, and make your configuration simple and discoverable. It really works great at that.

I do this for our services, it works great and we can easily put SSO in front of them with CF Access. I publish a Docker container that you can use as a sidecar for your Compose deployments:

https://gitlab.com/stavros/docker-cloudflared

I use this with Harbormaster (https://gitlab.com/stavros/harbormaster) so I can expose containerized stuff without ever forwarding any ports outside of Docker.

I use Dokku for that (I can share my Bitwarden repo if you want, the entire thing is four lines or something). I also made https://gitlab.com/stavros/harbormaster for things that weren't so "web server -> app -> database" and love it.

I had the same problem and didn't want to manage things by hand, so I wrote Harbormaster:

https://gitlab.com/stavros/harbormaster

It basically pulls Compose apps from the git repositories you specify, builds the containers and makes sure they're running. Pretty simple and works really well for me.

(This post should read "Argo tunnel" instead of just "Argo")

I did the same to enable secure access to services via SSO at work. I used Harbormaster[1] to deploy Compose files, but it's otherwise the same setup.

One of the big advantages this has is that the services can't be accessed any other way (not even from the same host, as they only listen inside the Docker network). That makes it hard to forget some port exposed because you listened to 0.0.0.0 instead of localhost.

Cloudflare access is very easy to set up SSO with, as well. I'd recommend this setup if you need it, though for home usage I usually just set up Caddy as a reverse proxy with basic auth, as I'll be the only person using this and I don't want Cloudflare MITMing my personal stuff.

[1]: https://gitlab.com/stavros/harbormaster

Something like harbormaster? https://gitlab.com/stavros/harbormaster