Write Your Own Task Queue

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • SonarQube - Static code analysis for 29 languages.
  • SaaSHub - Software Alternatives and Reviews
  • miniqueue

    A simple, single binary, message queue. Supports HTTP/2 and Redis Protocol.

    Interesting, I took this to heart a couple of years ago and came up with something with a serious focus on simplicity within a niche.

    https://github.com/tomarrell/miniqueue

  • bb-remote-execution

    Tools for Buildbarn to allow remote execution of build actions

    Though it obviously depends on the case at hand, I sort of agree with this.

    For a distributed build cluster that I maintain (Buildbarn, https://github.com/buildbarn/bb-remote-execution/), I also had to implement a scheduler process that would queue compilation/test actions, so that they can be executed on workers later on.

    Initially I looked into using some conventional queueing system, but eventually settled on implementing my own as part of the scheduler process. So far I'm really happy with this choice, as it has allowed me to implement the following features, and more:

    - In-flight deduplication of identical compilation actions. If identical actions are scheduled with different priorities, the highest priority is used.

    - Multi-level scheduling fairness between groups, users in a group, builds run by the same user, etc.. The fairness cooperates well with priorities.

    - Automatic removal of queued actions that are no longer associated with any running build.

    - Stickiness, where workers prefer picking up actions that are similar to the one they ran previously, for reducing network utilisation.

    - Facilities for draining workers.

    Though I'm not saying it would have been impossible to achieve this with an off the shelf task queue, I'm not convinced it would have been easy. Adding new features right now only means I need to care about the actual semantics of it, as opposed to trying to figure out how to map it onto the feature set of the queueing system of choice.

  • InfluxDB

    Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.

  • workerpool-go

    auto-scaling worker pool (work queue), using Go 1.18 generics

  • asciiflow

    ASCIIFlow

  • BeanstalkD

    Beanstalk is a simple, fast work queue.

    The only task queue I loved was beanstalkd -- it's beautifully written and highly performant. Starting it takes seconds and it's been running for a decade:

    https://beanstalkd.github.io/

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts