conductor VS dslabs

We celebrated a remarkable milestone in September when the Netflix Conductor GitHub repository reached 10k stars. It was a momentous achievement for our DevRel team. Just a month later, we're thrilled to announce that we've surpassed 12k stars! ⭐🎉

Also, don’t forget to give us a ⭐ on our Netflix Conductor repo.

One of my favorite workflow engines that has a really simple way to do things was not listed here, so I'll call it out - Netflix Conductor (https://github.com/Netflix/conductor).

Its capabilities comes to light when you model really complex workflows and one real value is how its all very visual not just during modeling but when running it. The history remains visible and you can even see how the whole flow evolved.

Yet another significant milestone on our journey: we've proudly reached the 10,000-star mark on our Netflix Conductor GitHub repository! 🌟

Give something like https://github.com/Netflix/conductor a try to solve this -- makes it very easy to do what you are trying to achieve.

Checkout Conductor https://github.com/Netflix/conductor which is far more scalable and easy on the resources with its own Celery like queues. Fully supports writing task workers in python:

I recently took a Distributed Systems course, and I also thought it was very interesting and unexpected how in the most basic form of Paxos, there is no concept of node roles or hierarchy like leader/follower, master/replica, etc. The base case is that all nodes have the same replicated log, and are "writers" capable of initiating changes to the log.

This youtube video was particularly helpful in learning about consensus algorithms, specifically Paxos/MultiPaxos:

https://www.youtube.com/watch?v=JEpsBg0AO6o

John Ousterhout (author of Raft) walks through Paxos/MultiPaxos as outlined by Leslie Lamport, and then talks about a series of optimizations to improve performance. One key optimization is transitioning from performing consensus on a single log-slot proposal, to the entire log-slot altogether, which mitigates failed consensus rounds, and is where concepts like leader nodes emerge. If your familiar with Raft and Paxos, while listening to these optimization applied to MultiPaxos, you can kind of notice it to begin to resemble Raft.

The course I took was through Georgia Tech, but was largely based around a framework developed at the University of Washington called dslabs:

https://github.com/emichael/dslabs

It was super informative for my learning about the foundations of distributed systems, namely consensus algorithms. I'd highly recommend it for anyone interested in learning more. Although fair warning, the programming assignments were quite difficult and time consuming.

I took a Distributed Systems course at Georgia Tech this spring, which used a learning framework from the University of Washington: https://github.com/emichael/dslabs

You make a key-value store using multiple techniques, from a basic single-node KV store, to a primary/replica, to PAXOS, to sharded PAXOS (which is essentially what AWS DynamoDB is)

There are tests to validate your implementation. I learned a ton from this, although I gave up at the last milestone because my grade was satisfactory in the class :)

If you open the syllabus from OMSCS site, it says that the 5 programming assignments will be based on https://github.com/emichael/dslabs

If you want to learn about distributed systems, this project is very cool imo. https://ellismichael.com/dslabs/ Don’t put your solution in a public repository.

Per the original author's repo:

This might interest you as well: https://github.com/emichael/dslabs

That distributed systems lab is what Georgia Tech's Distributed System lab[0] is based on, at least when I took the course back in 2021

[0] - https://omscs.gatech.edu/cs-7210-distributed-computing

Haven't taken the course, so not sure on specifics, but the projects are generally based on dslabs, as highlighted in the syllabus for the course.

If the former, I don't want to discourage you, but good luck. In my MS we had to build an extremely simplified PAXOS implementation and it took everyone 200+ hrs and I think 1 or 2 out of 50 students ended with a fully correct implementation. Actually, that project is publicly available, here: https://github.com/emichael/dslabs

fwiw this is used in many graduate programs around the country to teach distributed systems concepts: https://github.com/emichael/dslabs and can be done completely independently.

DSLabs: https://github.com/emichael/dslabs is the base of the projects and cumulates in implementing a Google Spanner inspired distributed KV store. Look through that repo and if you think you can handle it, you’ll probably be fine.