命令
启动zookeeper
启动两台机器上的kafaka
nohup kafka-server-start.sh $KAFKA_HOME/config/server.properties &
这里的
启动kafka的消费端
kafka-console-consumer.sh --zookeeper hadoop01:2181 --topic test
启动flume端(kafka是两个机器都要启动,flume只要启动一个机器上的就行。注意kafka和flume的区别)
flume-ng agent -n a1 -c conf -f $FLUME_HOME/conf/exec.conf -Dflume.root.logger=INFO,console
其中exec.conf配置如下
a1.sources = r1 a1.channels = c1 a1.sinks = k1 a1.sources.r1.type = exec a1.sources.r1.command = tail -F /export/data/flume_sources/click_log/1.log a1.sources.r1.channels = c1 a1.channels.c1.type=memory a1.channels.c1.capacity=10000 a1.channels.c1.transactionCapacity=100 a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink #这里
org.apache.flume.sink.kafka.KafkaSink 对应的是flume里的一个jar包a1.sinks.k1.topic = testa1.sinks.k1.brokerList = hadoop01:9092a1.sinks.k1.requiredAcks = 1a1.sinks.k1.batchSize = 20a1.sinks.k1.channel = c1
然后 echo "fdfsafjasdlfjlas" >>/export/data/flume_sources/click_log/1.log
观察 消费端的变化