Build a data pipeline on AWS with Kafka, Kafka connect and DynamoDB

• kafka-connect-datagen

  3 172 7.2 Java

  Connector that generates data for demos

  

The first part will keep things relatively simple - it's all about get started easily. I will be using the Kafka Connect Datagen source connector to pump data some sample data into MSK topic and then use the AWS DynamoDB sink connector to persist that data in a DynamoDB table.

• aws-msk-iam-auth

  4 130 6.5 Java

  Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.

  

sudo -u ec2-user -i mkdir iam-auth && cd ./iam-auth wget https://github.com/aws/aws-msk-iam-auth/releases/download/1.1.0/aws-msk-iam-auth-1.1.0-all.jar cd ../ cat < /home/ec2-user/kafka/config/client-config.properties # Sets up TLS for encryption and SASL for authN. security.protocol = SASL_SSL # Identifies the SASL mechanism to use. sasl.mechanism = AWS_MSK_IAM # Binds SASL client implementation. sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required; # Encapsulates constructing a SigV4 signature based on extracted credentials. # The SASL client bound by "sasl.jaas.config" invokes this class. sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler EOF export CLASSPATH=/home/ec2-user/iam-auth/aws-msk-iam-auth-1.1.0-all.jar echo "export CLASSPATH=${CLASSPATH}" | tee -a ~/.bash_profile

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project