Benthos - Fancy stream processing made operationally mundane

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • Benthos

    Fancy stream processing made operationally mundane

  • Benthos is a free and open source data streaming engine written entirely in Go and packaged as a static single-binary command line tool. It comes with a wide range of connectors and is totally data agnostic.

  • franz-go

    franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.6+. Producing, consuming, transacting, administrating, etc.

  • If you find the kafka input slow, try kafka_franz. It might be a bit faster, since it’s based on https://github.com/twmb/franz-go. The kafka one is based on https://github.com/Shopify/sarama. You can also write a custom input based on https://github.com/confluentinc/confluent-kafka-go, but this library relies on CGo, which can be annoying.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • sarama

    Discontinued Sarama is a Go library for Apache Kafka. [Moved to: https://github.com/IBM/sarama]

  • If you find the kafka input slow, try kafka_franz. It might be a bit faster, since it’s based on https://github.com/twmb/franz-go. The kafka one is based on https://github.com/Shopify/sarama. You can also write a custom input based on https://github.com/confluentinc/confluent-kafka-go, but this library relies on CGo, which can be annoying.

  • Confluent Kafka Golang Client

    Confluent's Apache Kafka Golang client

  • If you find the kafka input slow, try kafka_franz. It might be a bit faster, since it’s based on https://github.com/twmb/franz-go. The kafka one is based on https://github.com/Shopify/sarama. You can also write a custom input based on https://github.com/confluentinc/confluent-kafka-go, but this library relies on CGo, which can be annoying.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts