Streaming data into Kafka S01/E04 — Loading Log files using Grok Expression

This page summarizes the projects mentioned and recommended in the original post on dev.to

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • kafka-connect-file-pulse

    🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

  • This is the fourth and last article of the series "Streaming data into Kafka" series. In the first three articles, we saw how it's fairly easy to use Kafka Connect to load records from CSV, XML and JSON files into Apache Kafka without writing a single line of code. For doing this, we have used the Kafka Connect FilePulse connector which packs with a lot of nice features to parse and transform data.

  • kafkacat

    Discontinued Generic command line non-JVM Apache Kafka producer and consumer [Moved to: https://github.com/edenhill/kcat]

  • Note: In the example above, we have used kafkacat to consume the topics. The option -o-1 is used to only consume the latest message

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • joni

    Java port of Oniguruma regexp library

  • Connect FilePulse brings the power of Grok Expression directly to Kafka Connect with the GrokFilter which under the hood uses the Joni library which is the Java port of Oniguruma regexp library. It also provides a lot of predefined and reusable grok patterns. See the complete list of patterns.

  • kafka-connect-transform-grok

    Grok Expression Transform for Kafka Connect.

  • Good news! we have externalized the work done with the GrokFilter to a dedicated to SMT called: Kafka Connect Grok Transformation.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Kafka Connect Filepulse 2.13.0 is now available! This version includes support for SFTP and Alibaba OSS. It also contains many bug fixes and improvements. 🚀

    1 project | /r/apachekafka | 15 Sep 2023
  • Download json and csv file from github repository with apache kafka

    1 project | /r/apachekafka | 29 Jul 2021
  • debezium VS quix-streams - a user suggested alternative

    2 projects | 7 Dec 2023
  • Difficulty configuring log4j when deploying code as plugin for an app

    1 project | /r/CodingHelp | 27 Oct 2023
  • Real-time Data Processing Pipeline With MongoDB, Kafka, Debezium And RisingWave

    3 projects | dev.to | 18 Jul 2023