automq VS pgmq

[1] Cloud-Init: https://github.com/canonical/Cloud-Init [2] AutoMQ: https://github.com/AutoMQ/automq [3] Introduction to Cloud-Init: https://cloudinit.readthedocs.io/en/latest/explanation/introduction.html#how-does-Cloud-Init-work

AutoMQ: 1.1.0 https://github.com/AutoMQ/automq/releases/tag/1.1.0-rc0

References [1] Cloud Disks are (Really!) Expensive: https://www.warpstream.com/blog/cloud-disks-are-expensive [2] Making Apache Kafka Serverless: Lessons From Confluent Cloud: https://www.confluent.io/blog/designing-an-elastic-apache-Kafka-for-the-cloud/#self-balancing-clusters [3] How AutoMQ addresses the disk read side effects in Apache Kafka: https://www.automq.com/blog/how-automq-addresses-the-disk-read-side-effects-in-apache-Kafka [4] Broker performance degradation caused by call of sendfile reading disk in network thread:https://issues.apache.org/jira/browse/Kafka-7504 [5] Is there anyway to activate auto scaling or some form of auto scaling with Strimzi? : https://github.com/orgs/strimzi/discussions/6635 [6] Introducing Confluent Cloud Freight Clusters: https://www.confluent.io/blog/introducing-confluent-cloud-freight-clusters/ [7] Resize a Dedicated Kafka Cluster in Confluent Cloud: https://docs.confluent.io/cloud/current/clusters/resize.html [8] Why is low latency important?:https://redpanda.com/guides/Kafka-performance/Kafka-latency [9] Public Benchmarks and TCO Analysis: https://www.warpstream.com/blog/warpstream-benchmarks-and-tco [10] AutoMQ: https://github.com/AutoMQ/automq [11] AWS EBS Multi-Attach: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes-multi.html [12] NVME reservations: https://docs.aws.amazon.com/ebs/latest/userguide/nvme-reservations.html [13] Nitro card for Amazon EBS: https://d1.awsstatic.com/events/Summits/reinvent2023/STG210_Behind-the-scenes-of-Amazon-EBS-innovation-and-operational-excellence.pdf [14] How AutoMQ addresses the disk read side effects in Apache Kafka: https://www.automq.com/blog/how-automq-addresses-the-disk-read-side-effects-in-apache-Kafka [15] Kafka is dead, long live Kafka: https://www.warpstream.com/blog/Kafka-is-dead-long-live-Kafka [16] GCP Regional Persistent Disk: https://cloud.google.com/compute/docs/disks/high-availability-regional-persistent-disk [17] Azure ZRS Disk: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-deploy-zrs?tabs=portal

I agree with your viewpoint. The crux of the matter is not whether to use tiered storage or not, but what trade-offs have been made in the specific storage architecture and what benefits have been gained. Here(https://github.com/AutoMQ/automq?tab=readme-ov-file#-automq-...) is a qualitative comparison chart of streaming systems including kafka/confluent/redpanda/warpstream/automq. This comparison chart does not have specific numerical comparisons, but purely based on their trade-offs at the storage level, I think this will be of some use to you.

you can send or read a single message at a time or as many as you want in a batch.

https://github.com/tembo-io/pgmq?tab=readme-ov-file#read-mes...

And lots of interesting extensions use it, like

https://github.com/tembo-io/pgmq

https://github.com/zombodb/zombodb

https://github.com/supabase/pg_jsonschema

Have you considered https://github.com/tembo-io/pgmq for the queue bit?

some notes about pgmq, https://github.com/tembo-io/pgmq, that is on this list. It is built as an extension in Postgres, which makes it compatible with all languages that have a Postgres driver.

There's no 'magic' to it, it uses existing Postgres features so all the performance and consistency guarantees of Postgres are to be expected. Easily gets to 10k+ concurrent reads and writes even on smaller sized Postgres instances, which is more than most applications need.

Newer project but there's no library needed. https://github.com/tembo-io/pgmq. They have a pretty simple SQL api similar to SQS. It's an extension though, so some cloud provider will not support it.