Kafka 2017 Update(1)Zookeeper Cluster and Kafka Cluster
Install Zookeeper Cluster on 3 Nodes
>wget http://mirrors.advancedhosters.com/apache/zookeeper/zookeeper-3.5.3-beta/zookeeper-3.5.3-beta.tar.gz
>tar xf zookeeper-3.5.3-beta.tar.gz
>sudo ln -s /home/ec2-user/tool/zookeeper-3.5.3 /opt/zookeeper-3.5.3
>sudo ln -s /opt/zookeeper-3.5.3 /opt/zookeeper
Add the working directory to the PATH
PATH=$PATH:/opt/zookeeper/bin
Prepare the Configuration file
>cp conf/zoo_sample.cfg conf/zoo.cfg
Try start local server
>zkServer.sh start conf/zoo.cfg
Port 8080 is used by the AdminServer which is new in 3.5.0.
Add the settings in the zkServer.sh
nohup "$JAVA" $ZOO_DATADIR_AUTOCREATE "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.admin.serverPort=8081"\
"-Dzookeeper.log.file=${ZOO_LOG_FILE}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
-XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='kill -9 %p' \
-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &
if [ $? -eq 0 ]
https://zookeeper.apache.org/doc/r3.5.1-alpha/zookeeperAdmin.html#sc_adminserver_config
Source code is here https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/admin/JettyAdminServer.java
It is reading from the JVM system configuration of -Dzookeeper.admin.serverPort=8081
Visit the Admin console
http://fr-stage-api:8081/commands/stats
Connect with Client
>zkCli.sh -server localhost:2181
Stop the Service
>zkServer.sh stop
Prepare the Configuration for Cluster zoo1.cfg, zoo2.cfg, zoo3.cfg, adding these lines
server.1=fr-stage-api:2888:3888
server.2=fr-stage-consumer:2888:3888
server.3=fr-perf1:2888:3888
>vi /tmp/zookeeper/myid
1
>vi /tmp/zookeeper/myid
2
>vi /tmp/zookeeper/myid
3
Copy the file to all other server and start them all.
Exception:
2018-01-02 19:38:03,370 [myid:] - ERROR [main:QuorumPeerMain@86] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing conf/zoo1.cfg
at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:138)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:110)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
Caused by: java.lang.IllegalArgumentException: myid file is missing
Solution:
https://github.com/31z4/zookeeper-docker/issues/13
On the system settings:
ZOO_MY_ID=1
export ZOO_MY_ID
This is not necessary if I directly install zookeeper on my local.
Check status on Server
>zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
Kafka Cluster
Kafka requires Gradle 2.0 or higher
Install Gradle Manually https://gradle.org/install/#manually
>wget https://downloads.gradle.org/distributions/gradle-4.4.1-bin.zip
>unzip gradle-4.4.1-bin.zip
>sudo ln -s /home/ec2-user/tool/gradle-4.4.1 /opt/gradle-4.4.1
>sudo ln -s /opt/gradle-4.4.1 /opt/gradle
>gradle --version
------------------------------------------------------------
Gradle 4.4.1
------------------------------------------------------------
Build time: 2017-12-20 15:45:23 UTC
Revision: 10ed9dc355dc39f6307cc98fbd8cea314bdd381c
Groovy: 2.4.12
Ant: Apache Ant(TM) version 1.9.9 compiled on February 2 2017
JVM: 1.8.0_60 (Oracle Corporation 25.60-b23)
OS: Linux 4.1.13-18.26.amzn1.x86_64 amd64
Run gradle command on the source directory of Kafka
>gradle
Build Jar package
>./gradlew jar
Build the release Jar Package
>./gradlew releaseTarGz -x signArchives
Copy the binary file out to install directory
>cp ./core/build/distributions/kafka_2.11-1.0.0.tgz ~/install/
Link that from the right working directory
>sudo ln -s /home/ec2-user/tool/kafka-1.0.0 /opt/kafka-1.0.0
Prepare Single Kafka Configuration
>cat config/server.properties
zookeeper.connect=fr-stage-api:2181,fr-stage-consumer:2181,fr-perf1:2181
Start the Server
>kafka-server-start.sh config/server.properties
Prepare the Cluster Configuration
>cp config/server.properties config/server1.properties
>cp config/server.properties config/server2.properties
>cp config/server.properties config/server3.properties
broker.id=1
broker.id=2
broker.id=3
Start the first kafka on the first machine
>nohup bin/kafka-server-start.sh config/server1.properties &
Exceptions:
[2018-01-03 18:17:02,882] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured broker.id 1 doesn't match stored broker.id 0 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
at kafka.server.KafkaServer.getBrokerIdAndOfflineDirs(KafkaServer.scala:615)
at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:92)
at kafka.Kafka.main(Kafka.scala)
Solution:
Clean the log directory
>rm -fr /tmp/kafka-logs/*
>nohup bin/kafka-server-start.sh config/server1.properties &
>nohup bin/kafka-server-start.sh config/server2.properties &
>nohup bin/kafka-server-start.sh config/server3.properties &
Create the topic
>bin/kafka-topics.sh --create --zookeeper fr-stage-api:2181,fr-stage-consumer:2181,fr-perf1:2181 --replication-factor 2 --partitions 2 --topic cluster1
Created topic "cluster1".
Producer
>bin/kafka-console-producer.sh --broker-list fr-stage-api:9092,fr-stage-consumer:9092,fr-perf1:9092 --topic cluster1
Consumer
>bin/kafka-console-consumer.sh --zookeeper fr-stage-api:2181,fr-stage-consumer:2181,fr-perf1:2181 --topic cluster1 --from-beginning
References:
Kafka 1~6
http://sillycat.iteye.com/blog/1563312
http://sillycat.iteye.com/blog/1563314
http://sillycat.iteye.com/blog/2015175
http://sillycat.iteye.com/blog/2015181
http://sillycat.iteye.com/blog/2094688
http://sillycat.iteye.com/blog/2108042
http://sillycat.iteye.com/blog/2215237
http://sillycat.iteye.com/blog/2183932
https://kafka.apache.org/downloads
zookeeper
http://sillycat.iteye.com/blog/2397642
http://sillycat.iteye.com/blog/2397645
Kafka 2017 Update(1)Zookeeper Cluster and Kafka Cluster
猜你喜欢
转载自sillycat.iteye.com/blog/2406569
今日推荐
周排行