chaosmonkey
oban
Our great sponsors
chaosmonkey | oban | |
---|---|---|
22 | 27 | |
14,490 | 3,043 | |
1.2% | - | |
2.0 | 9.3 | |
4 months ago | about 19 hours ago | |
Go | Elixir | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
chaosmonkey
-
Zero Downtime Postgres Upgrades
Never saw this communicated by Google, but Netflix is the company I have in mind for doing that: https://github.com/Netflix/chaosmonkey
-
Introduction to Chaos Engineering
In 2010 Netflix developed a tool called "Chaos Monkey", whose goal was to randomly take down compute services (such as virtual machines or containers), part of the Netflix production environment, and test the impact on the overall Netflix service experience. In 2011 Netflix released a toolset called "The Simian Army", which added more capabilities to the Chaos Monkey, from reliability, security, and resiliency (i.e., Chaos Kong which simulates an entire AWS region going down). In 2012, Chaos Monkey became an open-source project (under Apache 2.0 license). In 2016, a company called Gremlin released the first "Failure-as-a-Service" platform. In 2017, the LitmusChaos project was announced, which provides chaos jobs in Kubernetes. In 2019, Alibaba Cloud announced ChaosBlade, an open-source Chaos Engineering tool. In 2020, Chaos Mesh 1.0 was announced as generally available, an open-source cloud-native chaos engineering platform. In 2021, AWS announced the general availability of AWS Fault Injection Simulator, a fully managed service to run controlled experiments.
- Pour one out for the Netflix admins right now
- [URGENT] Netflix Engineer Needs Help Scaling Kubernetes Deployment or I'm Toast!
-
Weird topic: Looking for ideas to make the sys admin job more competitive and inject some adrenaline
Have a look at it https://github.com/Netflix/chaosmonkey
- It Took Just Four Days From Elon Gleefully Admitting Heβd Unplugged A Server Rack For Twitter To Have A Major Outage
-
Manager does a little code cleanup...
Itβs kinda like Chaos Monkey except Elon is the chaos
- Discussion Thread
-
This perfect plan is based on a rather childish assumption that no one will fight back
they really did lmao
-
What's the dumbest thing you have done since working in IT?
Thanks, that was an interesting read. Chaos Monkey
oban
-
How to Use Flume in your Elixir Application
Oban, backed by PostgreSQL or SQLite, also provides a queue-based job processing system. Exq, on the other hand, is backed by Redis. It provides features similar to Flume, but without built-in rate limiting and batch processing capabilities.
-
Postgres as Queue
In Elixir land Oban[0] uses Postgres as queue and seems to work quite well.
[0] - https://github.com/sorentwo/oban
-
Zero Downtime Postgres Upgrades
I hear you on that, and can say that Postgres is incredibly capable at going beyond typical relational database workloads. One example are durable queues that are transactionally consistent with the rest of the database play a unique role in our architecture that would otherwise require more ceremony. More details here: https://getoban.pro
We are also working on shifting some workloads off of Postgres on to more appropriate systems as we scale, like logging. But we intentionally chose to minimize dependencies by pushing Postgres further to move faster, with migration plans ready as we continue to reach new levels of scale (e.g. using a dedicated log storage solution like elastic search or clickhouse).
- Deno Cron
-
Switching to Elixir
You can actually have "background jobs" in very different ways in Elixir.
> I want background work to live on different compute capacity than http requests, both because they have very different resources usage
In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).
This means that you can actually hold incoming HTTP connections a lot longer without troubles.
A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).
As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.
> because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.
Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.
A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.
It handles retries, errors etc.
But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.
You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.
It allows to use the "right tool for the job" quite easily.
It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).
-
Show HN: A simple API/CLI for scheduling HTTP requests
Hi HN!
This is something I've been tinkering on for the past couple months. It's basically just an API/CLI for scheduling delayed or recurring jobs as HTTP requests.
I initially built it as a personal tool to save myself a bit of time on little side projects where I've needed scheduled/recurring alerts, but decided it could be a good opportunity to practice building out a nice landing page [0] and documentation [1]. And who knows, maybe someone else will find it useful Β―\_(γ)_/Β―
The tool relies heavily on Elixir's Oban [2] library for managing jobs, and Mintlify [3] for documentation. I also shamelessly stole most of the frontend design from Resend [4] because I'm a fan of the aesthetic and thought it would be good for my design chops to use their design as a guide. I also discovered Radix [5] UI while working on this, which ended up being immensely helpful for moving quickly on the frontend.
Anyways, I almost certainly spent a bit too much time on small UX details that are most likely utterly inconsequential, but it was a fun exercise in polish :)
All feedback is welcome!
[0] https://www.booper.dev/
[1] https://docs.booper.dev/
[2] https://github.com/sorentwo/oban
[3] https://mintlify.com/
[4] https://resend.com/
[5] https://www.radix-ui.com/
- Choose Postgres Queue Technology
-
Pg_later: Asynchronous Queries for Postgres
Idk about pgagent but any table is a resilient queue with the multiple locks available in pg along with some SELECT pg_advisory_lock or SELECT FOR UPDATE queries, and/or LISTEN/NOTIFY.
Several bg job libs are built around native locking functionality
> Relies upon Postgres integrity, session-level Advisory Locks to provide run-once safety and stay within the limits of schema.rb, and LISTEN/NOTIFY to reduce queuing latency.
https://github.com/bensheldon/good_job
> |> lock("FOR UPDATE SKIP LOCKED")
https://github.com/sorentwo/oban/blob/8acfe4dcfb3e55bbf233aa...
-
Keep the Monolith, but Split the Workloads
> Bad code in a specific part of the codebase bringing down the whole app, as in our November incident.
This is a non-issue if you're using a Elixir/Erlang monolith given its fault tolerant nature.
The noisy neighbour issue (resource hogging) is still something you need to manage though. If you use something like Oban[1] (for background job queues and cron jobs), you can set both local and global limits. Local being the current node, and global the cluster.
Operating in a shared cluster (vs split workload deployments) give you the benefit of being much more efficient with your hardware. I've heard many stories of massive infra savings due to moving to an Elixir/Erlang system.
1. https://github.com/sorentwo/oban
- Library for reliably running jobs
What are some alternatives?
litmus - Litmus helps SREs and developers practice chaos engineering in a Cloud-native way. Chaos experiments are published at the ChaosHub (https://hub.litmuschaos.io). Community notes is at https://hackmd.io/a4Zu_sH4TZGeih-xCimi3Q
broadway - Concurrent and multi-stage data ingestion and data processing with Elixir
chaos-mesh - A Chaos Engineering Platform for Kubernetes.
exq - Job processing library for Elixir - compatible with Resque / Sidekiq
WLED - Control WS2812B and many more types of digital RGB LEDs with an ESP8266 or ESP32 over WiFi!
Rihanna - Rihanna is a high performance postgres-backed job queue for Elixir
litmus - A fast python HTTP server inspired by japronto written in rust.
kafka_ex - Kafka client library for Elixir
room-assistant - Presence tracking and more for automation on the room-level
verk - A job processing system that just verks! π§β
hyperion.ng - The successor to Hyperion aka Hyperion Next Generation
honeydew - Job Queue for Elixir. Clustered or Local. Straight BEAM. Optional Ecto. πͺπ