Apache Camel VS ideas2

"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas:

- https://camel.apache.org/

- https://www.windmill.dev/

- https://github.com/huginn/huginn

Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.

Apache Camel Apache Nifi Spring Cloud

This reminds me more of Apache Camel[0] than other things it's being compared to.

> The process initiator puts a message on a queue, and another processor picks that up (probably on a different service, on a different host, and in different code base) - does some processing, and puts its (intermediate) result on another queue

This is almost exactly the definition of message routing (ie: Camel).

I'm a bit doubtful about the pitch because the solution is presented as enabling you to maintain synchronous style programming while achieving benefits of async processing. This just isn't true, these are fundamental tradeoffs. If you need a synchronous answer back then no amount of queuing, routing, prioritisation, etc etc will save you when the fundamental resource providing that is unavailable, and the ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. If you go into this ad hoc, and build in a leaky abstraction that asynchronous things are are actually synchronous and vice versa, before you know it you are going to have unstable behaviour or even worse, deadlocks all over your system and the worst part - the true state of the system is now hidden in which messages are pending in transient message queues everywhere.

What really matters here is to fundamentally design things from the start with patterns that allow you to be very explicit about what needs to be synchronous vs async (building on principles of idempotency, immutability, coherence, to maximise the cases where async is the answer).

The notion of Apache Camel is to make all these decisions a first class elements of your framework and then to extract out the routing layer as a dedicated construct. The fact it generalises beyond message queues (treating literally anything that can provide a piece of data as a message provider) is a bonus.

[0] https://camel.apache.org/

Since you're writing a Java app to consume this, I highly recommend Apache Camel to do the consuming of messages for it. You can trivially aim it at file systems, message queues, databases, web services and all manner of other sources to grab your data for you, and you can change your mind about what that source is, without having to rewrite most of your client code.

For a simple sequential Pipeline, my goto would be Apache Camel. As soon as you want complexity its either Apache Nifi or a micro service architecture.

🐪 Apache Camel : Camel JBang, A JBang-based Camel app for easily running Camel routes.

Apache Camel

So, my advice is this. Expand your knowledge. Pursue higher education on topics you are familiar with, but also explore topics you are not. Read documentation, but question it. I just found out about something called Apache Camel today that I am excited to read up on. Why is it better than Spring? Is it really? What's happening here? This is always what excites me as a developer and engineer. There is so much to learn.

My blogging/journalling setup is simple.

I just use GitHub. I just rely on the default repository view on GitHub.com

I create a README.md and add markdown headings to the bottom or to the top (bottom if its a journal, top if it's a blog) and then when I get to 100-800 I create a new repository and repeat.

https://github.com/samsquire/ideas (2013)

https://github.com/samsquire/ideas4

https://github.com/samsquire/ideas3

https://github.com/samsquire/ideas2

Thanks for posting this Ask HN question.

I journal ideas and thoughts about computers and software. I am interested in software architecture, parallelism, async, coroutines, database internals, programming language implementation, software design and the web.

https://github.com/samsquire/ideas (2013)

https://github.com/samsquire/ideas2

https://github.com/samsquire/ideas3

https://github.com/samsquire/ideas4 <-- this is recent but needs editing

https://github.com/samsquire/ideas5 <-- this is what I'm working on now

https://github.com/samsquire/startups

https://github.com/samsquire/blog <-- thoughts I want to write about, but incomplete

I use README.md on GitHub and create a heading at the bottom for each entry. I use Typora on Windows or the GitHub web interface to edit.

[3]: https://github.com/samsquire/ideas2#5-open-demand-mapping-an...

Thanks for this.

I love the idea of breaking up a flow into separately scheduled but still linear message flow.

I wrote about a similar idea in ideas2

https://github.com/samsquire/ideas2#84-communication-code-sl...

The idea is that I enrich my code with comments and a transpiler schedules different parts of the code to different machines and inserts communication between blocks.

I read about how Zookeeper algorithm for transactionality and robustness to messages being dropped, which is interesting reading.

https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals....

How does Mats compare?

LMAX disruptor has a pattern where you split up each side of an IO request into two events, to avoid blocking in an handler. So you would always insert a new event to handle an IO response.

I am more likely to journal and blog if the friction to creating a post is as simple as opening a document and writing. The important part of journalling or note software is that you actually create notes. I did use Hetzner to run a Wordpress blog but it had an overhead of server expenses and keeping Wordpress up-to-date.

I don't want my data trapped in a proprietary system where it is difficult to export, so I use plaintext. I looked into Publii [1] but I prefer my current plaintext setup. Today I journal software ideas, computer ideas, startup ideas and community ideas on GitHub in the open, as README.md files. My journal is all public on GitHub at the following links. There are over 550+ journal entries, I am sure you shall enjoy them.

https://github.com/samsquire/ideas

https://github.com/samsquire/ideas2

https://github.com/samsquire/ideas3

https://github.com/samsquire/ideas4

https://github.com/samsquire/startups

https://getpublii.com/