elasticmq VS parquet4s

Since a majority of services rely on the same system services, we can leverage Docker to start all the needed project services prior running services with the offline command. The template ships with a base docker-compose.yml that stands up an instance of ElasticMQ to emulate Amazon SQS's API. The Docker image can be extended to include other services like DynamoDB, Redis, or PostgresQL and can be run using yarn infrastructure:build or yarn infrastructure:start depending on if it's your first run or not.

however...we have banned use of SQS. unlike S3/Minio, there's nothing API-compatible that we can run offline. (well, there's one thing, but it's in-memory / single node only and doesn't give us the durability/reliability we want with SQS). instead we use RabbitMQ because it's identical at the API layer no matter where we are. AWS has a hosted option for this but we haven't switched to it, because our existing self-managed Rabbit clusters are stable and well-understood.

I don't have experience with cloud, but we used https://github.com/mjakubowski84/parquet4s for storing data to HDFS with Akka Streams. It uses Hadoop libraries for actual writing so it supports various object stores like S3 and Google Cloud Storage.