dtm VS temporal

- using a distributed transaction framework (like https://github.com/dtm-labs/dtm)

But in this article, we introduce a workflow pattern in github.com/dtm-labs/dtm. Under this pattern, a mixture of XA, SAGA and TCC can be applied to different branches in a single distributed transactions, allowing users to customize most of the contents of a distributed transaction, providing great flexibility.

The open source distributed transaction framework https://github.com/dtm-labs/dtm has a two-stage message pattern inside that handles this problem very well. The following is an example of the use of an interbank transfer operation.

The above Sub-transaction Barrier technique, when used in conjunction with the distributed transaction framework https://github.com/dtm-labs/dtm, has been made available in several language SDKs, with the following example code in Go.

Maybe you can take a look at github.com/dtm-labs/dtm which provide built-in TCC pattern

Distributed XA transactions can solve the above business problem. This article presents a solution based on dtm-labs/dtm. DTM is a popular distributed transaction framework which supports XA, Saga, OutBox, and TCC patterns.

Github: https://github.com/dtm-labs/dtm

This article gives an example of implementing a distributed transaction across multiple store engines, Mysql, Redis and Mongo. This example is based on the Distributed Transaction Framework https://github.com/dtm-labs/dtm and will hopefully help to solve your problems in data consistency across microservices.

I don't know if I agree with the argument regarding durability vs elastic execution. If I can get both (with a nice API/DX) via something like Temporal (https://github.com/temporalio/temporal), what's the drawback here?

Link to GitHub -->

However, as we mentioned, each shard needs management. Part of the management includes a cache of Workflow histories for that shard. We can see the History pods’ memory usage is rising quickly. If the pods run out of memory, Kubernetes will terminate and restart them (OOMKilled). This causes Temporal to rebalance the shards onto the remaining History pod(s), only to then rebalance again once the new History pod comes up. Each time you make a scaling change, be sure to check that all Temporal pods are still within their CPU and memory requests—pods frequently being restarted is very bad for performance! To fix this, we can bump the memory limits for the History containers. Currently, it is hard to estimate the amount of memory a History pod is going to use because the limits are not set per host, or even in MB, but rather as a number of cache entries to store. There is work to improve this: github.com/temporalio/temporal/issues/2941. For now, we’ll set the History memory limit to 8GB and keep an eye on them—we can always raise it later if we find the pod needs more.

Correct, the workflow's guarantee to always complete executing independent of hardware failures is dependent on the database not losing data. You host your workflow code with Temporal's Worker library, which talks to an instance of the Temporal Server [1], which is an open-source set of services (hosted by you or by Temporal Cloud), backed by Cassandra, MySQL, or Postgres. [2] So for instance increasing Cassandra's replication factor increases your resilience to disk failure.

[1] https://github.com/temporalio/temporal

[2] https://docs.temporal.io/clusters#persistence

Re:graph frameworks - thanks for the pointers, hadn't heard about them! I'd heard of temporal which I believe provides a similar memoization capability with the purpose of not losing work in workflows that failed partway through?

Documentation

Personally I am leveraging the knowledge and momentum of Replay to dive into the Python SDK, build out a couple of applications to deepen my knowledge around Workflows, Activities, and metrics, and continue inhaling knowledge via the monthly meetup, the application development guide, and documentation. By next year I’ll experience the conference, not as one new to Temporal, but as an expert—maybe even as one of the people helping with the architecture review or running a Birds of a Feather; if anything, I know I look forward to seeing YOU at next year’s event!