flume安装以及应用

下载

//wget http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz

wget http://mirrors.hust.edu.cn/apache/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz

解压

tar –zxvf apache-flume-1.8.0-bin.tar.gz

mv apache-flume-1.8.0-bin /usr/local/flume

3) 设置环境变量

Vim /etc/profile.d/flume.sh

Source /etc/profile

4) 配置java_home

cp flume-env.sh.template flume-env.sh

vim flume-env.sh

可视化:
--------------
   1.
   2.
   3.
   4.
   5.
package com.it18zhang.callloggen;

import java.util.HashMap;
import java.util.Map;

/**
*
*/
public class App {
public static Map<String,String> caller = new HashMap<String, String>();
static{
caller.put("15810092493", "史玉龙");
caller.put("18000696806", "赵贺彪");
caller.put("15151889601", "张倩 ");
caller.put("13269361119", "王世昌");
caller.put("15032293356", "张涛");
caller.put("17731088562", "张阳");
caller.put("15338595369", "李进全");
caller.put("15733218050", "杜泽文");
caller.put("15614201525", "任宗阳");
caller.put("15778423030", "梁鹏");
caller.put("18641241020", "郭美彤");
caller.put("15732648446", "刘飞飞");
caller.put("13341109505", "段光星");
caller.put("13560190665", "唐会华");
caller.put("18301589432", "杨力谋");
caller.put("13520404983", "温海英");
caller.put("18332562075", "朱尚宽");
caller.put("18620192711", "刘能宗");
}

public static void main(String[] args) {
genCallLog();
}

public static void genCallLog(){

}

生成jar包，部署到centos执行
--------------------------------
   1.使用maven生成jar文件
       ...
   2.部署到centos

   3.执行
       $>mkdir /home/centos/calllog
       java -cp calllog.jar com.it18zhang.callloggen.App /home/logs/calllog/calllog.log

   4.创建centos上的执行脚本
       [calllog.sh]
       #!/bin/bash
       java -cp calllog.jar com.it18zhang.callloggen.App /home/logs/calllog/calllog.log

   5.修改权限
       $>chmod a+x calllog.sh

   6.执行脚本
       $>cd ~/calllog
       $>./calllog.sh

手机号
---------------
   0755-67568979
   186----------

086+0755-67568979
固话
--------------
0755-67568979
0755-67568979

网络电话
-------------
12358757575765656565

启动zk集群[s201 + s202 + s203]
-------------
$>zkServer.sh start

启动kafka集群[s202 + s203 + s204]
-----------------
$>cd /soft/kafka/config
$>kafka-server-start.sh -daemon server.properties

创建kafka主题
-----------------
//创建主题
$>kafka-topics.sh --zookeeper s202:2181 --topic calllog --create --replication-factor 3 --partitions 4

//查看主题列表
$>kafka-topics.sh --zookeeper s202:2181 --list

//启动控制台消费者,消费calllog主题,用于测试.
$>kafka-console-consumer.sh --zookeeper s201:2181 --topic calllog

topic consumer均在zookeeper上

s201上编写flume配置文件件，实时收集calllog.log日志
--------------------------------------------
1.配置文件
[/soft/flume/conf/calllog.conf]
a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type=exec
#-F 最后10行,如果从头开始收集 -c +0 -F:持续收集后续数据,否则进程停止。
a1.sources.r1.command=tail -F -c +0 /home/centos/calllog/calllog.log

a1.channels.c1.type=memory

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = calllog
a1.sinks.k1.kafka.bootstrap.servers = s202:9092 s203:9092 s204:9092
a1.sinks.k1.kafka.flumeBatchSize = 20
a1.sinks.k1.kafka.producer.acks = 1

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

   2.启动flume收集程序
       $>./flume-ng agent -f /home/appinstall/installed/apache-flume-1.8.0-bin/conf/calllog.conf -n a1 &

在s202主机安装flume软件

flume例子：

flume安装以及应用

猜你喜欢