Streaming data into Kafka S01/E04 — Loading Log files using Grok Expression

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • Scout APM - Truly a developer’s best friend
  • Zigi - Delete the most useless function ever: context switching.
  • Sonar - Write Clean Java Code. Always.
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • kafka-connect-file-pulse

    🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

    This is the fourth and last article of the series "Streaming data into Kafka" series. In the first three articles, we saw how it's fairly easy to use Kafka Connect to load records from CSV, XML and JSON files into Apache Kafka without writing a single line of code. For doing this, we have used the Kafka Connect FilePulse connector which packs with a lot of nice features to parse and transform data.

  • kafkacat

    Generic command line non-JVM Apache Kafka producer and consumer [Moved to: https://github.com/edenhill/kcat]

    Note: In the example above, we have used kafkacat to consume the topics. The option -o-1 is used to only consume the latest message

  • Scout APM

    Truly a developer’s best friend. Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.

  • joni

    Java port of Oniguruma regexp library

    Connect FilePulse brings the power of Grok Expression directly to Kafka Connect with the GrokFilter which under the hood uses the Joni library which is the Java port of Oniguruma regexp library. It also provides a lot of predefined and reusable grok patterns. See the complete list of patterns.

  • kafka-connect-transform-grok

    Grok Expression Transform for Kafka Connect.

    Good news! we have externalized the work done with the GrokFilter to a dedicated to SMT called: Kafka Connect Grok Transformation.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts