How to Install Confluent Enterprise and Confluent Control Center
How to Install Confluent Enterprise and Confluent Control Center
Confluent Control Center is a web-based tool for managing and monitoring Apache Kafka. Control Center 3.3 provides the functionality for building and monitoring production data pipelines and streaming applications.
Data Streams
You can use Control Center to monitor your data streams end to end, from producer to consumer. Use Control Center to verify that every message sent is received (and received only once), and to measure system performance end to end. Drill down to better understand cluster usage, and identify any problems.System Health
Control Center can monitor the health of your Kafka clusters. You can see trends for important broker and topic health metrics, as well as set alerts on important cluster KPIs.Kafka Connect Configuration
You can also use Control Center to manage and monitor Kafka Connect: the open source toolkit for connecting external systems to Kafka. You can easily add new sources to load data from external data systems and new sinks to write data into external data systems. Additionally, you can manage, monitor, and configure connectors with Confluent Control Center.This guide explains the concepts, how to install, and how to use Control Center.
Prerequisite
- Confluent Platform Enterprise Edition is installed.
Install Confluent Platform
Confluent Enterprise:
# yum install confluent-platform-2.11 -y
[root@kafkaserver ~]# yum install confluent-platform-2.11 -y
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
* epel: d2lzkl7pfhq30w.cloudfront.net
* epel-debuginfo: d2lzkl7pfhq30w.cloudfront.net
Installed:
confluent-platform-2.11.noarch 0:3.3.3-1
Dependency Installed:
confluent-control-center.noarch 0:3.3.3-1 confluent-control-center-fe.noarch 0:3.3.3-1
confluent-kafka-connect-replicator.noarch 0:3.3.3-1 confluent-rebalancer.noarch 0:3.3.3-1
confluent-support-metrics.noarch 0:3.3.3-1
Complete!
[root@kafkaserver ~]#
[root@kafkaserver ~]# yum install confluent-platform-2.11 -y
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
* epel: d2lzkl7pfhq30w.cloudfront.net
* epel-debuginfo: d2lzkl7pfhq30w.cloudfront.net
Installed:
confluent-platform-2.11.noarch 0:3.3.3-1
Dependency Installed:
confluent-control-center.noarch 0:3.3.3-1 confluent-control-center-fe.noarch 0:3.3.3-1
confluent-kafka-connect-replicator.noarch 0:3.3.3-1 confluent-rebalancer.noarch 0:3.3.3-1
confluent-support-metrics.noarch 0:3.3.3-1
Complete!
[root@kafkaserver ~]#
For more info on How to install Confluent Platform Enterprise, Please follow this blog.
Configure Kafka and Kafka Connect
Before starting Control Center, you must update the Kafka and Kafka Connect configuration files.1. Uncomment these lines in the Kafka server configuration file (/etc/kafka/server.properties). This enables Confluent Metrics Reporter.
##################### Confluent Metrics Reporter #######################
# Confluent Control Center and Confluent Auto Data Balancer integration
#
# Uncomment the following lines to publish monitoring data for
# Confluent Control Center and Confluent Auto Data Balancer
# If you are using a dedicated metrics cluster, also adjust the settings
# to point to your metrics kakfa cluster.
metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=localhost:9092
#
# Uncomment the following line if the metrics cluster has a single broker
confluent.metrics.reporter.topic.replicas=1
[root@kafkaserver ~]# vi /etc/kafka/server.properties
# Confluent Control Center and Confluent Auto Data Balancer integration
#
# Uncomment the following lines to publish monitoring data for
# Confluent Control Center and Confluent Auto Data Balancer
# If you are using a dedicated metrics cluster, also adjust the settings
# to point to your metrics kakfa cluster.
metric.reporters=io.confluent.metrics.reporter.ConfluentMetricsReporter
confluent.metrics.reporter.bootstrap.servers=localhost:9092
#
# Uncomment the following line if the metrics cluster has a single broker
confluent.metrics.reporter.topic.replicas=1
[root@kafkaserver ~]# vi /etc/kafka/server.properties
2. Add these lines to the Kafka Connect properties file (/etc/kafka/connect-distributed.properties). This will add support for the interceptors.
# Interceptor setup
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
[root@kafkaserver ~]# vi /etc/kafka/connect-distributed.properties
[root@kafkaserver ~]#
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
[root@kafkaserver ~]# vi /etc/kafka/connect-distributed.properties
[root@kafkaserver ~]#
3. Start Confluent Platform.
$ confluent start
[root@kafkaserver ~]# confluent start
This CLI is intended for development only, not for production
https://docs.confluent.io/current/cli/index.html
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
[root@kafkaserver ~]#
[root@kafkaserver ~]# confluent start
This CLI is intended for development only, not for production
https://docs.confluent.io/current/cli/index.html
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
[root@kafkaserver ~]#
Configure and Start Control Center
1. Define the number of partitions (<num-partitions>) and replication (<num-replication>) settings for Control Center by adding these lines to your properties file (/etc/confluent-control-center/control-center.properties).
# Quick start partition and replication values
confluent.controlcenter.internal.topics.partitions=1
confluent.controlcenter.internal.topics.replication=1
confluent.controlcenter.command.topic.replication=1
confluent.monitoring.interceptor.topic.partitions=1
confluent.monitoring.interceptor.topic.replication=1
confluent.metrics.topic.partitions=1
confluent.metrics.topic.replication=1
$ control-center-start /etc/confluent-control-center/control-center.properties
[root@kafkaserver ~]# control-center-start /etc/confluent-control-center/control-center.properties
[2020-02-28 19:30:22,453] WARN [main] Invalid value 1 for configuration confluent.controlcenter.internal.topics.replication: Value must be at least 3 (io.confluent.controlcenter.ControlCenterConfig)
[2020-02-28 19:30:22,455] INFO [main] ControlCenterConfig values:
confluent.controlcenter.internal.topics.retention.bytes = -1
[root@kafkaserver ~]# control-center-start /etc/confluent-control-center/control-center.properties
[2020-02-28 19:30:22,453] WARN [main] Invalid value 1 for configuration confluent.controlcenter.internal.topics.replication: Value must be at least 3 (io.confluent.controlcenter.ControlCenterConfig)
[2020-02-28 19:30:22,455] INFO [main] ControlCenterConfig values:
confluent.controlcenter.internal.topics.retention.bytes = -1
You can navigate to the Control Center web interface at http://localhost:9021/.
Setup stream monitoring
Now that you have all of the services running, you can start building a data pipeline. As an example, you can create a small job to create data.1. Open an editor and enter the following text , and save this as totail.sh.
cat <<EOF >> totail.sh
#!/usr/bin/env bash
file=totail.txt
while true; do
echo This is just to say >> \${file}
echo >> \${file}
echo I have eaten >> \${file}
echo the plums >> \${file}
echo that were in >> \${file}
echo the icebox >> \${file}
echo >> \${file}
echo and which >> \${file}
echo you were probably >> \${file}
echo saving >> \${file}
echo for breakfast >> \${file}
echo >> \${file}
echo Forgive me >> \${file}
echo they were delicious >> \${file}
echo so sweet >> \${file}
echo and so cold >> \${file}
sleep 1
done
EOF
#!/usr/bin/env bash
file=totail.txt
while true; do
echo This is just to say >> \${file}
echo >> \${file}
echo I have eaten >> \${file}
echo the plums >> \${file}
echo that were in >> \${file}
echo the icebox >> \${file}
echo >> \${file}
echo and which >> \${file}
echo you were probably >> \${file}
echo saving >> \${file}
echo for breakfast >> \${file}
echo >> \${file}
echo Forgive me >> \${file}
echo they were delicious >> \${file}
echo so sweet >> \${file}
echo and so cold >> \${file}
sleep 1
done
EOF
2. Start this script. It writes the poem to /totail.txt once per second. Kafka Connect is used to load that into a Kafka topic.
Run chmod to grant user execution permissions.
[root@kafkaserver ~]# chmod u+x totail.sh
Run the script.
[root@kafkaserver ~]# bash +x totail.sh
3. Use the Kafka Topics tool to create a new topic:Run the script.
[root@kafkaserver ~]# bash +x totail.sh
[root@kafkaserver ~]# kafka-topics --zookeeper localhost:2181 --create --topic poem \
--partitions 1 --replication-factor 1
Created topic "poem".
[root@kafkaserver ~]#
--partitions 1 --replication-factor 1
Created topic "poem".
[root@kafkaserver ~]#
4. From the Control Center web interface http://localhost:9021/, click on the Kafka Connect button on the left side of the web interface. On this page you can see a list of sources that have been configured - by default it will be empty. Click the New source button.
5. From the Connector Class drop-down menu select FileStreamSourceConnector. Specify the Connection Name as Poem File Source. Once you have specified a name for the connection a set of other configuration options will appear.
6. In the General section specify the file as /totail.txt and the topic as poem.
Check more Kafka Articles:
KafkaManager Error: Yikes! Ask timed out
Kafka Manager OutOfMemoryError : java heap space?
Updated: How to Configure kafka Monitoring with JMX Exporter and Prometheus.
Kafka Manager OutOfMemoryError : java heap space?
Updated: How to Configure kafka Monitoring with JMX Exporter and Prometheus.
8. Create a new sink.
1. From the Kafka Connect tab, click the Sinks tab and then New sink.
2. From the Topics drop-down list, choose poem and click Continue.
3. In the Sinks tab, set the Connection Class to FileStreamSinkConnector, specify the Connection Name as Poem File Sink, and in the General section specify the file as sunk.txt.
4. Click Continue, verify your settings, and then Save & Finish to apply the new configuration.
Now that you have data flowing into and out of Kafka, let’s monitor what’s going on!
9. Click the Data Streams tab and you will see a chart that shows the total number of messages produced and consumed on the cluster. If you scroll down, you will see more details on the consumer group for your sink. Depending on your machine, this chart may take a few minutes to populate.
About Manas Ranjan Tripathy
No comments