optimus VS cadence

You can check odpf github, they created some dataops tools using go, one of the example is optimus (https://github.com/odpf/optimus) which is a data pipeline orchestrator

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

this may help: https://github.com/odpf/optimus

Uber | Software Engineers | Hybrid (Denmark) | https://www.uber.com/dk/en/careers/locations/aarhus/

Work with an amazing team responsible for the infrastructure software that makes Uber’s data centers around the world reliable and scalable. If you want to solve the toughest engineering challenges alongside some of the smartest people in the industry, Uber Aarhus is the right place for you.

The team in Aarhus build and operate the stateless and stateful compute platforms used by nearly every other engineer in the company (Up - https://www.uber.com/en-GB/blog/up-portable-microservices-re... and Odin - https://www.uber.com/en-GB/blog/how-uber-optimized-cassandra...) as well as other related infrastructure projects such as Cadence - https://github.com/uber/cadence.

May be this..https://cadenceworkflow.io/

a managed esb orchestration for example is exactly same as step functions and workflow engines like cadence - https://github.com/uber/cadence

Having done a reasonable amount of messaging code in my time, I would say the final form of this sort of thing might look more like Cadence[0] than anything like this.

[0] https://github.com/uber/cadence

Emit events from your primary DB (postgres, etc.) to something like kafka or rabbitmq and then catch that in your search engine. There's also some end-to-end solutions like temporal (temporal.io) or cadence (https://cadenceworkflow.io/)