Our great sponsors
-
pubsub
This repository contains open-source projects managed by the owners of Google Cloud Pub/Sub. (by GoogleCloudPlatform)
-
kinesis-kafka-connector
kinesis-kafka-connector is connector based on Kafka Connect to publish messages to Amazon Kinesis streams or Amazon Kinesis Firehose.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
In this example, we’re reading from the Kafka topic demo-event-stream-export and writing to the GCP PubSub topic mux-demo-event-stream-export in the project external-mux using the GCP PubSub sink implementation. The connector will run 6 Kafka Connector tasks; each task works as a Kafka consumer, so it makes sense for the number of tasks to not exceed the number of partitions on the source topic (e.g. demo-event-stream-export). The connector sink is identified by the connector.class setting, which references the GCP sink implementation.
This example shows a connector that writes to a Kinesis sink using the AWS Kinesis sink implementation. Again, it’s reading from the Kafka topic demo-event-stream-export. There are Kafka Connect standard configuration keys (e.g. name, topics, connector.class, tasks.max) intermingled with the sink configuration keys (e.g. streamName, roleARN, ttl).
In our case, we add sink connectors for AWS Kinesis and Google Cloud PubSub. We also add a Prometheus exporter JAR that scrapes the Kafka Connect JMX metrics and exposes them as Prometheus metrics.