【Flume学习之二】Flume 使用场景

环境
　　apache-flume-1.6.0

一、多agent连接

1、node101配置 option2

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1

    # Describe/configure the source
    a1.sources.r1.type = netcat
    a1.sources.r1.bind = node101
    a1.sources.r1.port = 44444

    # Describe the sink
    # a1.sinks.k1.type = logger
    a1.sinks.k1.type = avro
    a1.sinks.k1.hostname = node102
    a1.sinks.k1.port = 60000

    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100

    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

2、node102配置 option1

############################################################
    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1

    # Describe/configure the source
    a1.sources.r1.type = avro
    a1.sources.r1.bind = node102
    a1.sources.r1.port = 60000

    # Describe the sink
    a1.sinks.k1.type = logger

    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100

    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1
    ############################################################

3、启动顺序
先启动node102-flume,后启动node101-flume，看一下flume启动顺序就知道,要先创建sink，然后创建channel,最后创建source；然后channel连接sink和channel;最后启动channel、sink、source

[root@node102 conf]# flume-ng agent -c /usr/local/apache-flume-1.6.0-bin/conf -f /usr/local/apache-flume-1.6.0-bin/conf/option1 -n a1 -Dflume.root.logger=INFO,console
[root@node101 conf]# flume-ng agent -c /usr/local/apache-flume-1.6.0-bin/conf -f /usr/local/apache-flume-1.6.0-bin/conf/option2 -n a1 -Dflume.root.logger=INFO,console

4、测试：在node101 telnet测试，在node102查看输出日志
node101 telnet:

[root@node101 ~]# telnet node101 44444
Trying 192.168.118.101...
Connected to node101.
Escape character is '^]'.
hello world
OK
haha wjy
OK
hi xiaoming
OK
^]
telnet> quit
Connection closed.
[root@node101 ~]#

node102 flume日志：

2019-06-29 00:43:46,022 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 0D hello world. }
2019-06-29 00:45:04,365 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 61 68 61 20 77 6A 79 0D haha wjy. }
2019-06-29 00:45:13,713 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 69 20 78 69 61 6F 6D 69 6E 67 0D hi xiaoming. }

【Flume学习之二】Flume 使用场景

猜你喜欢