the-algorithm-ml VS Async Ruby

Blog summarising the change: https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm

Twitter's announcement | Main GitHub Repo | ML GitHub Repo | Engineering Blog Post

They have made a major part of the code source available recently: the algorithm. However there are at least three issues with calling this an "open source Twitter":

link

Yeah, that's a basic problem, plus the neural network in the middle is just a recipe for a neural network, without any of the training weights or anything, because it is the data, people's personal data, that this would embody, that is the source of wealth for the network.

Repo for their recommendation-engine: https://github.com/twitter/the-algorithm-ml

The Async gem is the natural successor, It's actively maintained, and allows you write synchronous code is if it wasn't non-blocking, and most libraries don't need any special support for Async (exceptions are gems with C extensions that do I/O and DB libraries with connection pooling that would otherwise be thread-based).

https://github.com/socketry/async uses coroutines and I think in general it’s been a great model with very few downsides in practice.

There's async I/O. Here's a library that leans on Ruby 3's fiber scheduler.

Hey folks, wanted to show this off and get feedback. Still early/experimental but there are quite a few concepts I'm excited about here. This project came about while writing a program in Go and loving its approach to concurrency. Being a long-time Rubyist I immediately started to think about what similar concepts might look like in Ruby.

I set out with two main design constraints:

1. Lightweight: I didn't want routines to be backed by fibers or threads. Having been involved some in the async project (https://github.com/socketry/async), I had some experience using fibers for concurrency but was curious if they could be avoided.

2. Explicitness: Routine behavior must be written to describe exactly how it is to behave. I always felt like concurrent code was hard to fully understand because of the indirection involved. On the spectrum between tedium and magical I wanted to err more on the side of tedium with Goru.

Goru routines are just blocks that are called once for every tick of the reactor. It is up to the developer to implement behavior in terms of a state machine, where on each tick the routine takes some action and then updates the state of the routine for the next tick. This fulfills both design constraints:

1. Because routines are just blocks, they weigh in at about ~345 bytes of memory overhead.

2. Routine behavior is explicit because it is written as a state machine inside the block.

Couple more features worth noting:

* Goru includes channels for buffered reading/writing (similar to channels in Go).

* Goru ships with primitives for non-blocking IO to easily build things like http servers.

Curious your thoughts!

For those who are interested here is the question.

Welcome! Ruby isn't exactly "dying", but the hype/popularity is definitely fading. This is primarily because Ruby is no longer "new", most of Ruby's popularity came from Rails, and now Rails is no longer the "new hotness". However, Ruby still has lots of awesome features and lots of awesome other libraries and frameworks, such as the new fancy irb gem that uses reline, nokogiri, chunky_png, the async gems, Dragon Ruby, SciRuby, Ronin, and the new Hanami web framework.