简介
kafka是一个分布式的基于生产者消费者的消息体系,具有高吞吐/低延时/高可用等特点,在异构系统集成/系统结偶/流处理/削峰等方面被广泛应用(结偶/异步/并行)
Kafka核心组件
Broker:一个Kafka实例
Topic:数据的逻辑集合
Producer:消息生产者
Consumer:消息消费者
Zookeeper:kafka集群的协调中心
搭建Kafka集群
1)规划节点:node01,node02,node03
2)下载kafka安装包
下载地址http://kafka.apache.org/downloads,选择Kafka版本
3)解压安装包到/home/hadoop/application/kafka目录下
>tar -xzvf kafka.tgz
4)创建软链接 ln -s /home/hadoop/application/kafka /usr/local/kafka
5)修改配置文件kafka-home/config/server.properties
#每个borker的id是唯一的,多个broker要设置不同的id
broker.id=0
#访问端口号
port=9092
#访问地址
host.name=node01
#允许删除topic
delete.topic.enable=true
#The number of threads handling network requests
num.network.threads=3
#The number of threads doing disk I/O
num.io.threads=8
#The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400
#The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400
#The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600
#存储数据路径,默认是在/tmp目录下,需要修改
log.dirs=/home/hadoop/application/kafka/data
#创建topic默认分区数
num.partitions=1
num.recovery.threads.per.data.dir=1
#数据保存时间,默认7天,单位小时
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
#zookeeper地址,多个地址用逗号隔开
zookeeper.connect=node01:2181,node02:2181,node03:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
6)创建数据存储目录/home/hadoop/application/kafka/data
7)拷贝到node02和node03节点
scp -r /home/hadoop/application/kafka hadoop@node02:~/application/kafka
scp -r /home/hadoop/application/kafka hadoop@node02:~/application/kafka
8)在node02和node03上创建软连接,参考4)
9)修改node02和node03配置文件到broker.id为1,2,节点名称为node02,node03
10)启动各个broker节点的server
nohup /home/hadoop/application/kafka/bin/kafka-server-start.sh ../config/server/properties &
常用命令
创建topic
kafka-topics.sh --create -zookeeper node01:2181 -replication-factor 2 -partitions 3 --topic demo
topic列表
kafka-topics.sh --list --zookeeper node01:2181
topic详情
kafka-topics.sh --describe --zookeeper node01:2181
修改topic分区
kafka-topics.sh --alter --zookeeper node01:2181 --topic demo --partitions 5
console生产者/消费者
kafka-console-consumer.sh --zookeeper node01:2181 --from-beginning --topic demo
kafka-console-producer.sh --broker-list node01:9092,ndoe02:9092,node03:9092 --topic demo