MSK Kafka Service Setup
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data.Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters. It lets you use Apache Kafka data-plane operations, such as those for producing and consuming data. It runs open-source versions of Apache Kafka. This means existing applications, tooling, and plugins from partners and the Apache Kafka community are supported without requiring changes to application code.
Ø Broker nodes — When creating an Amazon MSK cluster, you specify how many broker nodes you want Amazon MSK to create in each Availability Zone. In the example cluster shown in this diagram, there’s one broker per Availability Zone. Each Availability Zone has its own virtual private cloud (VPC) subnet.
Ø ZooKeeper nodes — Amazon MSK also creates the Apache ZooKeeper nodes for you. Apache ZooKeeper is an open-source server that enables highly reliable distributed coordination.
Ø Producers, consumers, and topic creators — Amazon MSK lets you use Apache Kafka data-plane operations to create topics and to produce and consume data.
Ø Cluster Operations You can use the AWS Management Console, the AWS Command Line Interface (AWS CLI), or the APIs in the SDK to perform control-plane operations. For example, you can create or delete an Amazon MSK cluster, list all the clusters in an account, view the properties of a cluster, and update the number and type of brokers in a cluster.
EC2 Configuration :
Ø Install Java : sudo yum install java-1.8.0
Ø Get Kafka : wget https://archive.apache.org/dist/kafka/2.2.1/kafka_2.12-2.2.1.tgz
Ø Extract Kafka : tar -xzf kafka_2.12–2.2.1.tgz
Ø Get Cluster ARN : aws kafka describe-cluster — cluster-arn “Cluster-Arn” — region
Ø Create Topic : bin/kafka-topics.sh — create — zookeeper “ZookeeperConnectString” — replication-factor 2 — partitions 1 — topic AWSKafkaTutorialTopic
Ø User the Trust store : cp /usr/lib/jvm/JDKFolder/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks
Ø client.properties : security.protocol=SSL ssl.truststore.location=/tmp/kafka.client.truststore.jks
Ø Get Brokers : aws kafka get-bootstrap-brokers — cluster-arn ClusterArn — region
Ø Producer : ./kafka-console-producer.sh — broker-list BootstrapBrokerStringTls — producer.config client.properties — topic AWSKafkaTutorialTopic
Consumer : ./kafka-console-consumer.sh — bootstrap-server BootstrapBrokerStringTls — consumer.config client.properties — topic AWSKafkaTutorialTopic — from-beginning
