How to do distributed cronjobs with worker queues?

• cron

  19 12,479 0.0 Go

  a cron library for go

I had a similar design challenge recently and I haven’t worked it out completely yet. But I needed to dynamically create CronJob a based on a request and run each job in a go routine. I landed on https://github.com/robfig/cron which supports that behavior and removing jobs as well. I see this as being similar to your need since each scheduled cron entry via this library is sorta like having a queue with each having their own possible schedule. But I suppose it’s totally different from what you’re looking for. Curious to watch replies here.

• gocron

  12 4,898 8.3 Go

  Easy and fluent Go cron scheduling. This is a fork from https://github.com/jasonlvhit/gocron

  

There is gocron which you would need to implement with a locking solution like in apscheduler perhaps with postgres or redis.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• BeanstalkD

  14 6,467 0.0 C

  Beanstalk is a simple, fast work queue.

  

• aurora

  1 589 10.0 JavaScript

  Cross-platform beanstalkd queue server admin console. (by xuri)

• proposals

  60 63 4.0

  Temporal proposals (by temporalio)

  

Use Temporal open source project. It is much more than a cronjob service. But it supports managing periodic jobs natively.

• Airflow

  169 34,397 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

Airflow might also be a good option for you. Essentially DAGs of cronjobs. We like it a lot.

• Dkron

  3 4,124 8.4 Go

  Dkron - Distributed, fault tolerant job scheduling system https://dkron.io

  

Works great. Hope this helps. (For easiness GH here).

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project