Autoscaling Redis applications on Kubernetes 🚀🚀

• keda

  91 7,791 9.5 Go

  KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

  

In this blog, you will learn how to automatically scale your Celery workers that use Redis as the broker. There are multiple ways to achieve this - this blog uses a Kubernetes-based Event Driven Autoscaler (KEDA) to do the heavy lifting, including scaling up the worker fleet based on workload and also scaling it back to zero if there are no tasks in the queue!

• Resque

  5 9,390 4.1 Ruby

  Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.

  

Redis Lists are quite versatile and used as the backbone for implementing scalable architectural patterns such as consumer-producer (based on queues), where producer applications push items into a List, and consumers (also called workers) process those items. Popular projects such as resque, sidekiq, celery etc. use Redis behind the scenes to implement background jobs.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• gocelery

  6 2,322 0.0 Go

  Celery Distributed Task Queue in Go

  

Please note that this blog post uses a Golang application (thanks to gocelery!) as an example, but the same applies to Python or any other application that uses the Celery protocol.

• redis-celery-kubernetes-keda

  1 7 3.2 Go

  Autoscale Redis applications on Kubernetes

git clone https://github.com/abhirockzz/redis-celery-kubernetes-keda cd redis-celery-kubernetes-keda

• charts

  28 15,373 2.1 Go

  Discontinued ⚠️(OBSOLETE) Curated applications for Kubernetes (by helm)

  

Redis: I have used Azure Cache for Redis, but feel free to explore other options e.g. you can install one in your Kubernetes cluster using a Helm chart).

• virtual-kubelet

  10 4,085 6.7 Go

  Virtual Kubelet is an open source Kubernetes kubelet implementation.

  

If this sounds interesting, do check out Virtual Nodes in Azure Kubernetes Service to see how you can use them to seamlessly scale your applications to Azure Container Instances and benefit from quick provisioning of pods, and only pay per second for their execution time. The virtual nodes add-on for AKS, is based on the open source project Virtual Kubelet which is an open source Kubernetes kubelet implementation.

• Redis

  319 64,893 9.7 C

  Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.

  

This blog post demonstrates how to auto-scale your Redis based applications on Kubernetes. Redis is a widely used (and loved!) database which supports a rich set of data structures (String, Hash, Streams, Geospatial), as well as other features such as pub/sub messaging, clustering (HA) etc. One such data structure is a List which supports operations such as inserts (LPUSH, RPUSH, LINSERT etc.), reads (LRANGE), deletes (LREM, LPOP etc.) etc. But that's not all!

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• kubernetes

  662 106,923 10.0 Go

  Production-Grade Container Scheduling and Management

  

Kubernetes cluster along with kubectl: I have used Azure Kubernetes Service, although minikube should work just as well.

Suggest a related project