s4 VS nitter

the idea of map reduce remains a good one.

there are a number of interesting innovations in streaming systems that followed, mostly around reducing latency, reducing batch size, and alternate failure/retry strategies.

even hadoop could be hard to debug when hitting a performance ceiling for challenging workloads. the streaming systems took this even further, spark being notorious for fiddle with knobs and pray the next job doesn’t fail after a few hours, again.

i played around with the thinnest possible map reduce stack a while back[1][2]. i wanted to understand the performance ceiling for different workloads without all the impenetrable layers of data bureaucracy. turns out modern network and cpu are really fast when you stop adding random software layers like lasagna.

i think the future of data, for serious workloads, is gonna be bespoke. the primitives are just too good now.

1. https://github.com/nathants/s4

2. https://github.com/nathants/bsv

pipes are great. is the other process on another cpu or another machine? honestly who cares.

https://github.com/nathants/s4/blob/master/examples/nyc_taxi...

i dragged my feet on go for a long time. i also thought that skipping go and moving to rust was the play. a few years later, i still write python often, but i don’t build systems with it. python i now use like bash, to glue things together and automate random things. it’s a fantastic language and i will never drop it.

the verbosity of go is the biggest hurdle for a pythonista. the thought of giving up context managers, decorators, iterators, comprehensions, exceptions, coroutines, it’s unthinkable. in comparison go is ugly. your aesthetic mind screams in protest.

write go full time. dive in. as months pass, not only will those aesthetic objections fade, your mental model from python cleanly transforms to go. go is what mypy tried to be. the cost was aesthetic changes. the benefit is worth it.

the zen of python says if it’s easy to explain it might be a good idea. this is go, and it is.

i rebuilt a reasonably sized project from python[1] to go[2] over the last few years. i also have a system that i maintained both python[3] and go[4] implementations for, sharing a test suite in python.

go, like python, is fantastic. use both in whatever amount works for you. don’t read about them, build with them. you won’t regret it.

1. https://github.com/nathants/cli-aws/tree/bb78e529e7d1d3f95ac...

2. https://github.com/nathants/libaws

3. https://github.com/nathants/s4/tree/python

4. https://github.com/nathants/s4

Don't use Youtube without going through a proxy like Invidious [1] or Newpipe

Don't use {site} Search without going through a proxy like SearxNG [2]

Don't use TwiXXer without going through a proxy like Nitter - this has gotten more difficult lately but it still works as long as you feed the daemon some registered accounts. Video does not work at the moment but that seems to be fixable.

Don't use Reddit without going through a proxy like libreddit [4]

Start noticing the pattern? Maybe it is time to start producing promotional posters:

The only thing to come between you and ADS could be a proxy / ADS. I'ts just not worth the risk

ADS / New rules for a sane net / Sane net protects you, your partner and your community

A proxy here and a filter there, ADS nowhere

The more you tighten your grip, ${site}, the more viewers will slip through your fingers

[1] https://github.com/iv-org/invidious

[2] https://github.com/searxng/searxng

[3] https://github.com/zedeus/nitter

[4] https://github.com/libreddit/libreddit

https://nitter.esmailelbob.xyz/liuzhuang1234/status/17601959...

(bit of trial and error from https://github.com/zedeus/nitter/wiki/Instances)

I assume this is why Nitter got slammed and is on the verge of breaking.

https://github.com/zedeus/nitter/issues/1154

The creator of the project left this comment an hour ago:

https://github.com/zedeus/nitter/issues/1155#issuecomment-19...

(HN removed the deep link to the specific comment from the title)

Most of the time you can just replace twitter.com with nitter.net or nitter.it but when these instances are down or blocked you can use other instances listed here: https://github.com/zedeus/nitter/wiki/Instances

But as the other commenter said Libredirect is the way to go: https://libredirect.github.io/

Use a Youtube proxy like Invidious [1], problem solved and you get to subscribe to channels without telling the Beast about your interests. Add Sponsorblock (which supports Invidious) to get rid of any in-stream advertising which remains and you'll be transported back to those hallowed times of yore when men were men, women were women and advertising was something you found in newspapers. Youtube will try to make this harder just like Xitter is trying to make it harder to use proxies like Nitter [2].

[1] https://github.com/iv-org/invidious

[2] https://github.com/zedeus/nitter

njal.la restored nitter.net, and replied with their reasons: https://github.com/zedeus/nitter/issues/1150#issuecomment-18...

tbh I can totally understand why they acted this way.