inline-java VS beam

Also, inline-java

If you want to interact with Java libraries, but don't need the Haskell program itself to run on the JVM, inline-java is probably your best bet at the moment.

will do the job, but due to a bug in versions prior to this commit the tag parameter will be ignored. The cached object is going to be reloaded even if you provide the same identifier, rendering the whole mechanism useless and our transformation will hit our attached resources every time.

Apache Beam a framework in which we can implement batch and streaming data processing pipeline independent of the underlying engine e.g. spark, flink, dataflow etc.

Apache Beam works this way. Code sample: https://github.com/apache/beam/blob/master/examples/multi-la...

ListItem(name='Apache Beam', website='https://beam.apache.org/', category='Batch Processing', short_description='Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream processing'),

Apache Beam

I asked a similar question in a different community, and the closest they came up with was the niche Apache Beam and the obligatory vague hand-waving about no-code systems. So, maybe DEV seeming to skew younger and more deliberately technical might get a better view of things? Is anybody using a "Framework of the Future" that we should know about?

Apache Beam may be what you're looking for. It will work with both Python and Java. It's used by GCP in the Cloud Dataflow service as a sort of streaming ETL tool. It occupies a similar niche to Spark, but is a little easier to use IMO.

Now that we understand how exactly-once state consistency works, you might think what about side effects, such as sending out an email, or write to database. That is a valid concern, because Flink's recovery mechanism are not sufficient to provide end to end exactly once guarantees even though the application state is exactly once consistent, for example, if message x and y from above contains info and action to send out email, during failure recovery, these messages will be processed more than once which results in duplicated emails being sent. Therefore other techniques must used, such as idempotent writes, transaction writes etc. To avoid the duplicated email issue, we could save the key of the message that have been processed to a key-value data storage, and use that to check the key, however, since steam processing means unbounded message, it is tricky to manage this key-value checker with large throughput or unbounded time window by yourself. Here I will explain one approach to guarantee end to end exactly once such as only sending out email once with Apache Beam.

No, Beam comes with many runners and can run in many environments, including locally. The Python SDK is the easiest to get running and here is the quickstart guide. It runs this simple wordcount script.

Dataflow is Google's implementation of a runner for Apache Beam jobs in Google cloud. Right now, python and java are pretty much the only two options supported for writing Beam jobs that run on Dataflow.