elasticsearch filebeat采集日志输出到elasticsearch
使用filebeat采集日志文件,将日志输出到logstash,logstash输出到elasticsearch
*****************************
日志文件:my.log
2020-01-12 10:06:14.549 INFO 9356 --- [main] com.example.demo.DemoApplication : Starting DemoApplication on LAPTOP-D73GD8TE with PID 9356 (started by 28401 in E:\java\IdeaProjects\springboot hello-world)
2020-01-12 10:06:14.555 INFO 9356 --- [main] com.example.demo.DemoApplication : No active profile set, falling back to default profiles: default
2020-01-12 10:06:16.775 INFO 9356 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2020-01-12 10:06:16.787 INFO 9356 --- [main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2020-01-12 10:06:16.787 INFO 9356 --- [main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.29]
2020-01-12 10:06:16.967 INFO 9356 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2020-01-12 10:06:16.967 INFO 9356 --- [main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 2314 ms
2020-01-12 10:06:17.422 INFO 9356 --- [main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2020-01-12 10:06:17.710 INFO 9356 --- [main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-01-12 10:06:17.781 INFO 9356 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2020-01-12 10:06:17.787 INFO 9356 --- [main] com.example.demo.DemoApplication : Started DemoApplication in 4.119 seconds (JVM running for 6.441)
2020-01-12 10:06:19.332 INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-01-12 10:06:19.333 INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2020-01-12 10:06:19.342 INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.s.web.servlet.DispatcherServlet : Completed initialization in 8 ms
2020-01-12 10:09:21.353 INFO 3104 --- [main] com.example.demo.DemoApplicationTests : Starting DemoApplicationTests on LAPTOP-D73GD8TE with PID 3104 (started by 28401 in E:\java\IdeaProjects\springboot hello-world)
2020-01-12 10:09:21.355 INFO 3104 --- [main] com.example.demo.DemoApplicationTests : No active profile set, falling back to default profiles: default
2020-01-12 10:09:23.758 INFO 3104 --- [main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2020-01-12 10:09:24.435 INFO 3104 --- [main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-01-12 10:09:24.516 INFO 3104 --- [main] com.example.demo.DemoApplicationTests : Started DemoApplicationTests in 3.886 seconds (JVM running for 5.747)
2020-01-12 10:09:25.677 INFO 3104 --- [SpringContextShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
*****************************
相关配置
filebeat配置
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
output.logstash:
hosts: ["172.18.0.32:5044"]
logstash管道配置
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["172.18.0.33:9200"]
index => "log-%{+yyyy.MM.dd}"
}
}
*****************************
docker 创建容器
docker run -it --net fixed --ip 172.18.0.31 \
-v /usr/elasticsearch/filebeat/config/filebeat3.yml:/usr/share/filebeat/filebeat.yml \
-v /usr/elasticsearch/filebeat/logs:/usr/share/filebeat/logs \
--name filebeat docker.elastic.co/beats/filebeat:7.5.1
docker run -it --net fixed --ip 172.18.0.32 -p 5044:5044 \
-v /usr/elasticsearch/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml \
-v /usr/elasticsearch/logstash/config/logstash3.conf:/usr/share/logstash/pipeline/logstash.conf \
--name logstash docker.elastic.co/logstash/logstash:7.5.1
docker run -it --net fixed --ip 172.18.0.33 -p 9201:9200 -p 9301:9300 \
-e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
-v /usr/elasticsearch/single/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
--name es-single2 elasticsearch:7.5.1
docker run -it --net fixed --ip 172.18.0.34 -p 5601:5601 \
-e ELASTICSEARCH_HOSTS="http://172.18.0.33:9200" \
--name kibana docker.elastic.co/kibana/kibana:7.5.1
*************************
kibana显示