RabbitMQ VS PostgreSQL

If you've ever worked as an enterprise developer in any moderately complex company, you've likely encountered distributed systems of the kind I want to talk about in this post—two or more systems communicating together via a message queue (MQ), such as RabbitMQ or Apache Kafka. Distributed, message-based systems are ubiquitous in today's programming landscape, especially due to the (now hopefully at least somewhat tempered) microservice architecture frenzy that swept over our field during the past decade.

RabbitMQ - Feature-rich message broker

Alternatives to: Redis (Queue, Pub/Sub), ZeroMQ, RabbitMQ, Apache Kafka, Amazon Simple Queue Service, Google Cloud Pub/Sub

When something important happens (like a new user signing up), that part of the app sends a message—think of it like dropping a letter in a mailbox. Other parts of the app can listen for those messages and react when they arrive. The middleman that passes the message along is called a message broker—and one of the most popular is RabbitMQ. This setup means the sender and the receiver don’t have to be active at the same time. One service can keep moving, and the rest will catch up.

RabbitMQ Official Documentation

rabbitmq

Asynchronous Communication Patterns: Utilize message queues and event-driven architectures to decouple services and improve scalability. By using these patterns, you can handle high volumes of asynchronous requests more efficiently. For example, implementing a message queue like RabbitMQ or Kafka can help manage the flow of requests between services.

Consume open-source queuing services – customers can deploy message brokers such as ActiveMQ or RabbitMQ, to develop asynchronous applications, and when moving to the public cloud, use the cloud providers managed services alternatives.

We are using redis as our broker. You can opt for RabbitMQ which is supported out-of-box by celery.

You're minding your own business, managing AWS infrastructure for a client with a pretty standard e-commerce setup: a Medusa.js backend, a Next.js storefront, and most importantly for this story, a PostgreSQL RDS instance safely stashed away in a private subnet where nothing from the outside world can touch it. Exactly how the AWS gods intended.

Your Database: This is your system's memory. It can be a fast in-memory store like Redis for temporary data (perfect for velocity checks) or a persistent relational database like PostgreSQL for long-term data (like blacklists).

#! /bin/bash set -ex IMG=postgres/test-0.0.1 IMG_ID=`docker images ${IMG} -q` PG_TAG=REL_18_BETA1 if [ "${IMG_ID}" = "" ]; then if [ ! -d "postgres-${PG_TAG}" ]; then wget https://github.com/postgres/postgres/archive/refs/tags/${PG_TAG}.tar.gz && tar -xzf ${PG_TAG}.tar.gz fi ID=991 USR=postgres USR_HOME=/home/postgres cat > Dockerfile << EOF FROM ubuntu:latest RUN groupadd -g ${ID} ${USR} && useradd -r -u ${ID} -g ${USR} ${USR} ADD postgres-${PG_TAG} ${USR_HOME} WORKDIR ${USR_HOME} RUN chown -R ${USR}:${USR} ${USR_HOME} RUN apt-get update && apt-get install -y g++ zlib1g-dev make curl tar gzip perl liblz4-dev libreadline-dev flex bison libicu-dev liburing-dev RUN apt-get install --reinstall -y pkg-config && ./configure --with-liburing --enable-debug --with-lz4 && make -j4 && make all && make install RUN echo "export PATH=/usr/local/pgsql/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" >> /etc/bash.bashrc && \ chown -R ${USR}:${USR} /usr/local/pgsql USER ${USR} EOF docker build -t ${IMG}:latest . rm Dockerfile rm -rf postgres-${PG_TAG} ${PG_TAG}.tar.gz else echo "Image ${IMG} already exists with ID ${IMG_ID}" fi

Choosing the database was straightforward: PostgreSQL. I have the most experience with it, very easy to spin up, has a lot of trust and is open-source (I guess the last two come hand-in-hand).

I was explaining why it is the way that it is. If you'd like your own version of a parser, here's Postgres' [0]. Personally, I really like SQL's syntax and find that it makes sense when reading it.

[0]: https://github.com/postgres/postgres/tree/master/src/backend...

If you’ve never tried PostgreSQL before, the official site is a great starting point: PostgreSQL: The world’s most advanced open source database.

/* https://github.com/postgres/postgres/blob/144ad723a4484927266a316d1c9550d56745ff67/src/backend/optimizer/path/costsize.c#L3375 */ void final_cost_nestloop(PlannerInfo *root, NestPath *path, JoinCostWorkspace *workspace, JoinPathExtraData *extra) { /* ... */ if (path->jpath.path.param_info) path->jpath.path.rows = path->jpath.path.param_info->ppi_rows; else path->jpath.path.rows = path->jpath.path.parent->rows; /* ... */ } /* https://github.com/postgres/postgres/blob/144ad723a4484927266a316d1c9550d56745ff67/src/backend/optimizer/path/costsize.c#L3873 */ void final_cost_mergejoin(PlannerInfo *root, MergePath *path, JoinCostWorkspace *workspace, JoinPathExtraData *extra) { /* ... */ if (path->jpath.path.param_info) path->jpath.path.rows = path->jpath.path.param_info->ppi_rows; else path->jpath.path.rows = path->jpath.path.parent->rows; /* ... */ } /* https://github.com/postgres/postgres/blob/144ad723a4484927266a316d1c9550d56745ff67/src/backend/optimizer/path/costsize.c#L4305 */ void final_cost_hashjoin(PlannerInfo *root, HashPath *path, JoinCostWorkspace *workspace, JoinPathExtraData *extra) { /* ... */ if (path->jpath.path.param_info) path->jpath.path.rows = path->jpath.path.param_info->ppi_rows; else path->jpath.path.rows = path->jpath.path.parent->rows; /* ... */ }

PostgreSQL as database

> As introduced in the previous chapters, all identifers are hashed, thus we can also hash the known keywords at startup and make comparing them very fast.

One trick that postgres uses [1][2] is perfect hashing [3]. Since you know in advance what your keywords are, you can design such hashing functions that for each w(i) in list of i keywords W, h(w(i)) = i. It essentially means no collisions and it's O(i) for the memory requirement.

[1] https://github.com/postgres/postgres/blob/master/src/tools/P...

[2] https://github.com/postgres/postgres/blob/master/src/tools/g...

[3] https://en.wikipedia.org/wiki/Perfect_hash_function

Understanding a database starts with understanding its structure. For PostgreSQL users, one of the most effective ways to visualize and manage your schema is by using an Entity-Relationship Diagram (ERD). Either if you're working with a large legacy database or starting something new, an ER diagram shows how your tables are connected and how your data is organized.