versatile-data-kit VS Benthos

Here's the project: https://github.com/vmware/versatile-data-kit

I don't really know if this would work for you because the UI is not functional yet, but a very simple REST API ingestion example here, there's one for csv too https://github.com/vmware/versatile-data-kit/wiki/Ingesting-data-from-REST-API-into-Database I can't imagine a simpler way unless it's really drag and drop.

I work at VMware and we use one tool for the whole ELT, it was made internally as there was no good alternative at the time and now we opensourced it, here it is: https://github.com/vmware/versatile-data-kit

With my team we created csv ingestion plugin described here, maybe you want to try it out: https://github.com/vmware/versatile-data-kit/wiki/Ingesting-local-CSV-file-into-Database

We use Versatile Data Kit for batch data job orchestration (https://github.com/vmware/versatile-data-kit)

"suggestions on how to reduce the time spent on initially generating and adjusting the code" is using some tools that automate ELT. Here's one open-source tool I'm working on with my team: https://github.com/vmware/versatile-data-kit

here's the repo: https://github.com/vmware/versatile-data-kit

If you wish to contribute, projects usually have good first issues: https://github.com/vmware/versatile-data-kit/labels/good%20first%20issue If you wish to learn, check out examples: https://github.com/vmware/versatile-data-kit/tree/main/examples

Have you heard about versatile data kit (https://github.com/vmware/versatile-data-kit)? I think it meets your needs perfectly:

Versatile Data Kit is a framework to bBuild, run and manage your data pipelines with Python or SQL on any cloud https://github.com/vmware/versatile-data-kit here's a list of good first issues: https://github.com/vmware/versatile-data-kit/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22 Join our slack channel to connect with our team: https://cloud-native.slack.com/archives/C033PSLKCPR

I’ve used https://www.benthos.dev/ and it’s really easy and well implemented. The author is also very responsive

Looks cool. What is the difference between this tools and benthos (https://www.benthos.dev/)?

You might want to gradually replace that one with https://github.com/twmb/franz-go because Shopify is looking to find a new owner for Sarama and, until or if they do, it seems to be falling behind with maintenance: https://github.com/Shopify/sarama/issues/2461 For example, they still haven’t addressed this breaking change https://github.com/Shopify/sarama/issues/2358. franz-go has worked well so far in Benthos https://github.com/benthosdev/benthos/tree/main/internal/impl/kafka and it will likely end up as the only implementation once the Sarama-based one will be deprecated

I upload a mix of code reviews and live streams on https://www.youtube.com/@Jeffail, mostly building https://www.benthos.dev out in the open so the content ranges from beginner friendly stuff to more advanced things like stream processing, parser combinators, etc.