s4 VS cmdg

the idea of map reduce remains a good one.

there are a number of interesting innovations in streaming systems that followed, mostly around reducing latency, reducing batch size, and alternate failure/retry strategies.

even hadoop could be hard to debug when hitting a performance ceiling for challenging workloads. the streaming systems took this even further, spark being notorious for fiddle with knobs and pray the next job doesn’t fail after a few hours, again.

i played around with the thinnest possible map reduce stack a while back[1][2]. i wanted to understand the performance ceiling for different workloads without all the impenetrable layers of data bureaucracy. turns out modern network and cpu are really fast when you stop adding random software layers like lasagna.

i think the future of data, for serious workloads, is gonna be bespoke. the primitives are just too good now.

1. https://github.com/nathants/s4

2. https://github.com/nathants/bsv

pipes are great. is the other process on another cpu or another machine? honestly who cares.

https://github.com/nathants/s4/blob/master/examples/nyc_taxi...

i dragged my feet on go for a long time. i also thought that skipping go and moving to rust was the play. a few years later, i still write python often, but i don’t build systems with it. python i now use like bash, to glue things together and automate random things. it’s a fantastic language and i will never drop it.

the verbosity of go is the biggest hurdle for a pythonista. the thought of giving up context managers, decorators, iterators, comprehensions, exceptions, coroutines, it’s unthinkable. in comparison go is ugly. your aesthetic mind screams in protest.

write go full time. dive in. as months pass, not only will those aesthetic objections fade, your mental model from python cleanly transforms to go. go is what mypy tried to be. the cost was aesthetic changes. the benefit is worth it.

the zen of python says if it’s easy to explain it might be a good idea. this is go, and it is.

i rebuilt a reasonably sized project from python[1] to go[2] over the last few years. i also have a system that i maintained both python[3] and go[4] implementations for, sharing a test suite in python.

go, like python, is fantastic. use both in whatever amount works for you. don’t read about them, build with them. you won’t regret it.

1. https://github.com/nathants/cli-aws/tree/bb78e529e7d1d3f95ac...

2. https://github.com/nathants/libaws

3. https://github.com/nathants/s4/tree/python

4. https://github.com/nathants/s4

Partly the reason I wrote and use this command line client for GMail: https://github.com/ThomasHabets/cmdg

Also became a fun learning experience about terminals.

https://github.com/ThomasHabets/cmdg

I wanted to use GMail from a fast cli that used the native gmail API.

https://github.com/ThomasHabets/rslurp

I wanted to download concurrently and according to patterns. Ok, so honestly this one probably exists somewhere in a form that I would like, but I couldn't find it.

https://github.com/ThomasHabets/sim

I wanted multi-party authorization for sudo, and couldn't find one.

https://github.com/ThomasHabets/monotonic_clock

People kept using gettimeofday, so this is part of my compaign against it. (see https://blog.habets.se/2010/09/gettimeofday-should-never-be-...)

https://github.com/ThomasHabets/gtping

I worked in mobile core networks, and wanted a "ping" that used the GTP protocol since that won't be firewalled.

https://github.com/ThomasHabets/ind

I wanted my bash scripts to have automatic indentation, while not sacrificing buffering latency and such.

https://github.com/ThomasHabets/tlscheck

I wanted a simple tool to audit my TLS certificates for expiry.

https://github.com/google/huproxy

I was travelling to China on vacation and wanted a VPN out that would be unlikely to be blocked by the great firewall. Ok, so there are many VPN-like tools for getting through the GFW. Maybe it was just an excuse for me to write it. Honestly ssh -D would have likely worked just fine. It's being used by the keymaster project now though, so maybe it did something right: https://github.com/Cloud-Foundations/keymaster/blob/master/d...

https://github.com/google/tcpauth

I wanted to lock down SSH to anyone who doesn't have a secret key (and portknocking is usually ridiculous). Why not use TCP MD5 for it? https://github.com/google/tcpauth

Also, aerc. Or something like cmdg for Gmail specifically.