Redis VS celery

Since 2008 I have been an official Debian Developer and even the Debian Project Leader from 2017—2019. I have submitted over 4,000 bugs to the project, been part of several core teams and have released over 250 security updates as well (as part of the Debian Long Term Support initiative). Today, I am the release manager of the Lintian static analysis tool but I also maintain a number of popular packages such as Redis, Django and Memcached.

I think that sums it up pretty well. The description of redis at https://redis.io/ is "Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker.". Without knowing anything about the application that you want to build, I personally wouldn't have good feeling to use Redis, that's defined as an "in-memory data structure" as the primary database for it - compared to a NoSQL database like MongoDB or a relational database like MySQL or PostgreSQL. I've been using Redis in projects as a cache (similar to memcached) besides the main database.

In-process caching may be implemented in a Node.js application through libraries, such as node-cache, memory-cache, api-cache, and others. There is a wide variety of distributed caching solutions, but the most popular ones are Redis and Memcached. They are both in-memory key-value stores and optimal for read-heavy workloads or compute-intensive workloads due to their use of memory rather than the slower on-disk storage mechanisms found in traditional database systems.

Redis stands for REmote DIctionary Server. It is an open source, fast NoSQL database written in ANSI C and optimized for speed. Redis is an in-memory database that means that all data in Redis is stored in RAM, delivering the fastest possible access times to the data for both read and write requests.

Here, have some cache

Redis is an in-memory key-value store which can save abstract data structures with high performance. The open-source software is typically used for database, messaging, and caching functions.

According to its GitHub repository, Redis (stands for REmote DIctionary Server) is an in-memory data structure store. It is a disk-persistent key-value database with support for multiple data structures such as strings, hashes, lists, sets, bitmaps, etc.

Redis offers highly performant and efficient read and write along with few operations mentioned in this article. There are lots more. For more information, you can visit the documentation.

If you need distributed access then redis

Would recommend checking out celery: https://github.com/celery/celery

Several years ago, finishing another such task at work and spending a lot of time integrating the Pytest with SQLAlchemy (I am loving TDD so much), I decided to create a boilerplate for such a service. At that time, I used Celery. So I stuffed my project with Celery, SQLAlchemy, Docker and Pytest.

If either shutdown (or close) raises an exception, self.sock isn't set to None. Thus Python retains the reference.

https://github.com/celery/celery/issues/4843#issuecomment-99...

Another way you can have leaks with a reference counting GC is through circular references. Python provides WeakRef for such situations, but it's up to the programmer to use it correctly.

Ideally try to avoid using Celery at all. It's the most consistently buggy, poorly documented, worst quality major Python project I have used so far.

Here is a fresh example: they deprecated CELERY_ prefixes in Django settings for some reason, which makes zero sense [1]. But because it's Celery, they only warn on a few properties but not others, and the tool they ship inexplicably renamed completely unrelated settings for me [2]. And yes, apparently the new way doesn't work either [3].

It's very characteristic of what I've seen from Celery over the last few years of working with it. If you are doing anything greenfield, it's best to avoid it altogether.

[1] https://stackoverflow.com/questions/65554242/celery-imports-...

[2] https://twitter.com/lambdadmitry/status/1468337594358546435?...

[3] https://github.com/celery/celery/issues/7140

Celery is used for background tasks, with Redis as the celery backend.

So when can this actually be used? Well, for our project, we have a Django web application project that implements Celery Asynchronous Tasks. We can schedule these tasks to be run by a background worker, but we require an audit log to determine who started a task and what organization the task is tied back to.

Have a worker or two (or more, depending on how many messages you're getting) who subscribe to this queue & all they do is take each upload notification and queue up a Celery task to process it. In your case, it's just going to queue up a task that takes the path to the video in Minio (and any other info like user id, or whatnot) and processes it.

Celery does not allow a rate limit per task and parameter (in our case, domain). Meaning that we can throttle workers or queues, but not to a fine-grained detail as we would like to. There are several issues open and workarounds. From reading several of those, the take-away is that we cannot do it without keeping track of the requests ourselves.

Figured it out using, https://github.com/celery/celery/issues/3759.

Looks like the solution to this in celery is to use the billiard library in place of multiprocessing. https://github.com/celery/celery/issues/4525