dozer VS plumber

https://github.com/getdozer/dozer/issues/1631#issuecomment-1...

and then something has gone off the rails about the accounting process since

  Trigger.dev

Hi HN,

I'm Matteo and co-founder of Dozer (https://github.com/getdozer/dozer), a backend tool for building data products. Over the past few months we have done good progress with the product development but our documentation needs a lot of work. Myself, my co-founder and the entire team are juggling among many activities, and one of them is producing good documentation, and that is the part where we are bit struggling.

We have tried to hire a couple of tech writers but it's really not working out as we wanted. Our product is a dev product, and it is very hard to find tech writers who have that depth to think what a developer need. At the same time, asking the dev team to write documentation produces low quality docs (obviously devs prefer to write code and they just to go back to that as fast they can).

The result is that I end up writing most of documentation or refactoring the entire content produced by tech writer. But this approach will not scale.

How do you guys solved it? I'm curios to know about the approach of other startups.

Thanks

This blog post explores how to enhance these aspects by implementing a low-latency cache using Dozer and PostgreSQL. We delve into the importance of caching in data management, the limitations of not having a cache, and the unique advantages of using Dozer as a cache layer. We also provide a guide on how to use Dozer for real-time data ingestion from PostgreSQL. This post is a must-read for developers seeking to improve their application's performance and responsiveness, and for those looking to simplify their data infrastructure.

I recently penned a blog post on an interesting topic - leveraging Dozer with PostgreSQL to create a high-speed, low-latency cache. In case you're not familiar with Dozer, it's an open-source data API backend built entirely in Rust, known for its fast processing and lower resource usage.

some time ago I shared about our OSS project Dozer with the group. We just did a presentation and demo at the Python Meetup group here in Singapore and just wanted to share the full video here.

Hey HN,

As one of the co-founders of Dozer (https://github.com/getdozer/dozer), I stumbled upon this awesome article from the Temporal folks (https://temporal.io/blog/saga-pattern-made-easy) and got super excited about the potential synergy between Dozer and Temporal.

While Temporal nails the orchestration of microservices, I couldn't help but notice the challenge of consolidating data across services. In the article, they talk about three services: flight booking, hotel booking, and experience booking. But imagine trying to create a complete trip view by stitching together data from all three services - it can be quite a headache!

And that's where Dozer swoops in to save the day. With Dozer, I can tap into all the change data capture (CDC) events from each microservice, merge the data in real-time, stash it in a cache, and make it available through APIs. The best part? Since Dozer operates in real-time, my cached data is always fresh. Boom! Now I can simply leverage Dozer's APIs to access a comprehensive trip view.

Raise your hand if you've faced this challenge before! Let's hear your thoughts.

👋🏼 Are you looking for a platform that simplifies the process of building and deploying data-driven applications? Look no further than Dozer! 💻 is a powerful Data API backend that simplifies the process of building and deploying data-driven applications & APIs for real-time data processing and analysis, predictive analytics, data caching, and much more. Dozer is a 100% open-source project.

Star our repository and get a chance to win an amazing Swagpack!!🎁🎉 We will announce the winners in our Discord server. So, don’t forget to join.🤗

You can consider using Dozer (https://github.com/getdozer/dozer), an open-source data API backend. Dozer has a PostgreSQL connector that you can use to monitor any changes in your database schema and data. Although Dozer is not specifically designed for ETL tasks, you can still use it for this purpose by using Lambda functions. Dozer provides a Lambda runtime to execute user-defined functions. So you can use PostgreSQL connector ( to capture CDC) with Lambda runtime to load data to snowflake.

Enter Dozer, it simplifies the process of creating low-latency data APIs (gRPC and REST) from any data source easily with OpenAPI specification. Dozer is a real-time data engineering tool built with Rust, which enables seamless integration of data streams(sources), efficient data processing, and lightning-fast response times. This allows developers to focus on building applications without worrying about the complexity of managing real-time data.

Streamdal

4. Go to the provided UI (or run the CLI app) and be able to peek into what your app is reading or writing, like with `tail -f`.

And that's basically it. There's a bunch more functionality in the project but we find this to be the most immediately useful part. Every developer we've shown this to has said "I wish I had this at my gig at $company" - and we feel exactly the same. We are devs and this is what we’ve always wanted, hundreds of times - a way to just quickly look at the data our software is producing in real-time, without having to jump through any hoops.

If you want to learn more about the "why" and the origin of this project - you can read about it here: https://streamdal.com/manifesto

— — —

HOW DOES IT WORK?

The SDK establishes a long-running session with the server (using gRPC) and "listens" for commands that are forwarded to it all the way from the UI -> server -> SDK.

The commands are things like: "show me the data that you are currently consuming", "apply these rules to all data that you produce", "inspect the schema for all data", and so on.

The SDK interprets the command and either executes Wasm-based rules against the data it's processing or if it's a `tail` request - it'll send the data to the server, which will forward it to the UI for display.

The SDK IS part of the critical path but it does not have a dependency on the server. If the server is gone, you won't be able to use the UI or send commands to the SDKs, but that's about it - the SDKs will continue to work and attempt to reconnect to the server behind the scenes.

— — —

TECHNICAL BITS

The project consists of a lot of "buzzwordy" tech: we use gRPC, grpc-Web, protobuf, redis, Wasm, Deno, ReactFlow, and probably a few other things.

The server is written in Go, all of the Wasm is Rust and the UI is Typescript. There are SDKs for Go, Python, and Node. We chose these languages for the SDKs because we've been working in them daily for the past 10+ years.

The reasons for the tech choices are explained in detail here: https://docs.streamdal.com/en/resources-support/open-source/

— — —

LAST PART

OK, that's it. What do you think? Is it useful? Can we answer anything?

- If you like what you're seeing, give our repo a star: https://github.com/streamdal/streamdal

Checkout plumber https://github.com/streamdal/plumber if you want all the Postgres changes sent to basically any type of broker queue https://docs.streamdal.com/en/data-ingestion/relay/postgresql-cdc/. I would say Nat's Jetstream is probably the way to go if you have K8s running already. It's a dead simple service written in Go. Just make sure you allocate enough memory to Jetstream.

I recommend Streamdal. The connecting agent is open source and distributed by default, so it will scale horizontally WAY better than Debezium. All data ingested is indexed into parquet as well, and you can do serverless functions/transforms on the platform to reduce Snowflake compute costs.

For CDC and event driven part of the stack, Plumber is a great free tool. That project is going to be adding sampling soon too - this can def help with the cost of ETL.

https://github.com/batchcorp/plumber check it out if you want to get into event driven systems

That’s awesome to see. My coworker and I always figured Go would be perfect for this. Going to be a serious amount of work! I see you use NATS as well. Big fan of it. Checkout our project https://github.com/batchcorp/plumber if you end up needing to inspect or send messages while deving against it.

Checkout plumber https://github.com/batchcorp/plumber join our slack https://launchpass.com/streamdal we got a pretty knowledgeable group