Streaming data into Kafka S01/E04 — Loading Log files using Grok Expression

• kafka-connect-file-pulse

  4 305 8.9 Java

  🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

  

This is the fourth and last article of the series "Streaming data into Kafka" series. In the first three articles, we saw how it's fairly easy to use Kafka Connect to load records from CSV, XML and JSON files into Apache Kafka without writing a single line of code. For doing this, we have used the Kafka Connect FilePulse connector which packs with a lot of nice features to parse and transform data.

• kafkacat

  8 3,573 7.3 C

  Discontinued Generic command line non-JVM Apache Kafka producer and consumer [Moved to: https://github.com/edenhill/kcat]

Note: In the example above, we have used kafkacat to consume the topics. The option -o-1 is used to only consume the latest message

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• joni

  1 162 5.8 Java

  Java port of Oniguruma regexp library

  

Connect FilePulse brings the power of Grok Expression directly to Kafka Connect with the GrokFilter which under the hood uses the Joni library which is the Java port of Oniguruma regexp library. It also provides a lot of predefined and reusable grok patterns. See the complete list of patterns.

• kafka-connect-transform-grok

  1 16 0.0 Java

  Grok Expression Transform for Kafka Connect.

  

Good news! we have externalized the work done with the GrokFilter to a dedicated to SMT called: Kafka Connect Grok Transformation.

Suggest a related project