Streaming data into Kafka S01/E04 — Loading Log files using Grok Expression

1. kafka-connect-file-pulse

   1 4 327 7.7 Java

   🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

   

   This is the fourth and last article of the series "Streaming data into Kafka" series. In the first three articles, we saw how it's fairly easy to use Kafka Connect to load records from CSV, XML and JSON files into Apache Kafka without writing a single line of code. For doing this, we have used the Kafka Connect FilePulse connector which packs with a lot of nice features to parse and transform data.

2. InfluxDB

   influxdata.com featured

   InfluxDB high-performance time series database. Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems.

   

3. kafkacat

   2 8 3,573 7.3 C

   Discontinued Generic command line non-JVM Apache Kafka producer and consumer [Moved to: https://github.com/edenhill/kcat]

   Note: In the example above, we have used kafkacat to consume the topics. The option -o-1 is used to only consume the latest message

4. joni

   3 1 172 7.7 Java

   Java port of Oniguruma regexp library

   

   Connect FilePulse brings the power of Grok Expression directly to Kafka Connect with the GrokFilter which under the hood uses the Joni library which is the Java port of Oniguruma regexp library. It also provides a lot of predefined and reusable grok patterns. See the complete list of patterns.

5. kafka-connect-transform-grok

   4 1 16 0.0 Java

   Grok Expression Transform for Kafka Connect.

   

   Good news! we have externalized the work done with the GrokFilter to a dedicated to SMT called: Kafka Connect Grok Transformation.

Suggest a related project