etsd VS docker-rollout

A pretty same setup with a bunch of differences:

1. I'm using a single postgresql database for all apps (each with a different user) on a different server; each app has a different db user

2. I use a minio instance for file/media uploads/serving

3. I mostly use nginx but i'm transitioning new apps to caddy because of automatic integration with let's encrypt and much smaller config for common purposes

4. I use a fab-classic (fabric 1x) script to deploy new versions: https://github.com/spapas/etsd/blob/master/fabfile.py

5. For backup I do a logical db backup once per day via cron (using a script similar to this https://spapas.github.io/2016/11/02/postgresql-backup/)

6. One memcache instance of all apps

7. Each app gets a redis instance (if redis is needed): https://gist.github.com/akhdaniel/04e4bb2df76ef534b0cb982c1d...

8. Use systemd for app control

Yes, you are right on that. If the server is compromised a malicious user may change the client-side code to add a backdoor and steal your private key when you unlock it. He'll be able to steal only the keys that are unlocked while the backdoor stays undetected (not all the data).

The ideal way to resolve that would be to change the service to an API and offer binaries with a correct signature so the user can check and make sure that they get the correct thing. Actually I tried writing the client binaries using electron (https://github.com/spapas/etsd/tree/master/client) but didn't have the time for that :(

You are rigth though, I've added a Risks section to warn for that thingie https://github.com/spapas/etsd/blob/master/README.md#risks

I created a CDK deploy that uses docker-rollout [1][2] to deploy highly available infrastructure to EC2 using only autoscaling groups. It is not super polished but it is a complete example, so it could be useful if you are considering hosting on EC2. Rolling out deploys involves updating one file on S3 and running one script.

Ironically after all that setup, I decided to give Linode with k8s a try [3] :-) (due to aws' high costs of egress and NAT gws / IPv4 tax on AWS, and the fact that some apps that I want to run are easier to deploy with helm).

More notes:

* I did try ECS and Fargate, which are nice, but also come with associated costs and a bunch of complexity. At that point, I rather spend time directly with k8s, which should make my localhost parity way higher, and hosting somewhere more affordable.

* I tried both Pulumi and Terraform. I have mixed feelings about them. I ended up using CDK because it _felt_ like the nicer development experience (except when CloudFormation fails and it kind of hides the reason why, sigh ... fishing for logs on CloudWatch is such a drag!).

* I tried to add some NACL rules since I ended up running the thing on a public VPC. I couldn't make it work but at that time I had already decided to host elsewhere so I left it like that :-). I did succeed on adding support for AWS WAF. Sadly, the cdk currently doesn't have high level support for WAF so it was not as nice to setup.

--

1: https://github.com/Wowu/docker-rollout

2: https://news.ycombinator.com/item?id=34690947

3: https://medium.com/@elliotgraebert/comparing-the-top-eight-m...

````

This way, Caddy will buffer the request and give 30 seconds for your new service to get online when you're deploying a new version.

Ideally, during deployment of a new version the new version should go live and healthy before caddy starts using it (and kills the old container). I've looked at https://github.com/Wowu/docker-rollout and https://github.com/lucaslorentz/caddy-docker-proxy but haven't had time to prioritize it yet.