Airflow VS Apache Camel

A few years ago, I opened a GitHub issue with Microsoft telling them that I think the .NET ecosystem needs its own equivalent of Apache Airflow or Prefect. Fast forward 'til now, and I still don't think we have anything close to these frameworks.

Check out taskinstance.py in the Airflow project, it's a well targeted file, it has only one main class TaskInstance and a few small supporting classes and functions. It is ~3000 lines long: https://github.com/apache/airflow/blob/main/airflow/models/taskinstance.py

If you have the spare capacity Apache Airflow is great for this.

💡 You can read more here.

Its a bit overkill but i use Airflow with local executor.

To learn more about it, I built a Data Pipeline that uses Apache Airflow to pull Elon Musk tweets using the Twitter API and store the result in a CSV stored in a MinIO (OSS alternative to AWS s3) Object Storage bucket.

Airflow

I am currently having a problem with trying to enable Azure OAuth to authenticate into our airflow instance. I have posted in countless other places trying to get answers so this is my next place I am trying. Here is the link to the discussion I posted within the airflow repo: https://github.com/apache/airflow/discussions/28098 but I will also do the liberty of posting it here as well. If anybody has any knowledge or can help I would greatly appreciate it as I have been dealing with this for over a month with no answers.

Airflow is really popular, started at Airbnb. Pros: huge community, super mature. Cons: generic workflow orchestration, not the best for handling only data, hard to scale and maintain.

This reminds me more of Apache Camel[0] than other things it's being compared to.

> The process initiator puts a message on a queue, and another processor picks that up (probably on a different service, on a different host, and in different code base) - does some processing, and puts its (intermediate) result on another queue

This is almost exactly the definition of message routing (ie: Camel).

I'm a bit doubtful about the pitch because the solution is presented as enabling you to maintain synchronous style programming while achieving benefits of async processing. This just isn't true, these are fundamental tradeoffs. If you need a synchronous answer back then no amount of queuing, routing, prioritisation, etc etc will save you when the fundamental resource providing that is unavailable, and the ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. If you go into this ad hoc, and build in a leaky abstraction that asynchronous things are are actually synchronous and vice versa, before you know it you are going to have unstable behaviour or even worse, deadlocks all over your system and the worst part - the true state of the system is now hidden in which messages are pending in transient message queues everywhere.

What really matters here is to fundamentally design things from the start with patterns that allow you to be very explicit about what needs to be synchronous vs async (building on principles of idempotency, immutability, coherence, to maximise the cases where async is the answer).

The notion of Apache Camel is to make all these decisions a first class elements of your framework and then to extract out the routing layer as a dedicated construct. The fact it generalises beyond message queues (treating literally anything that can provide a piece of data as a message provider) is a bonus.

[0] https://camel.apache.org/

For a simple sequential Pipeline, my goto would be Apache Camel. As soon as you want complexity its either Apache Nifi or a micro service architecture.

🐪 Apache Camel : Camel JBang, A JBang-based Camel app for easily running Camel routes.

Sounds like Camel.

Camel

If you need to stick with spreadsheets I agree with the suggestion of Python--or Java, I used to automate a bunch of business processes with Apache Camel.

Or if you are looking for something light-weighter you can always use Camel camel.apache.org

apache camel dependency on spring boot, Kafka etc, also other libs dependencies Note: Kafka 1.1: https://mvnrepository.com/artifact/org.apache.camel/camel-kafka/2.22.4 kafka 2.0 : https://mvnrepository.com/artifact/org.apache.camel/camel-kafka/2.23.1 https://github.com/apache/camel/blob/master/components/camel-kafka/src/main/docs/kafka-component.adoc