安装
- 官网下载压缩包
https://www.elastic.co/cn/products/elasticsearch - 解压
tar -xzf logstash-7.5.1.tar.gz
- 新建logstash.conf文件,用于配置输入、过滤规则、输出信息
[root@m5315392269-5 logstash]# cd logstash-7.5.1/
[root@m5315392269-5 logstash-7.5.1]# touch logstash.conf
#logstash.conf文件中的内容
input {
beats {
type => "log"
port => "5044" #开始本机的5044端口,监听
}
}
filter{
mutate{
split=>["message","|"]
add_field => {
"log_date" => "%{[message][0]}"
}
add_field => {
"log_level" => "%{[message][1]}"
}
add_field => {
"log_thread" => "%{[message][2]}"
}
add_field => {
"log_class" => "%{[message][3]}"
}
add_field => {
"log_content" => "%{[message][4]}"
}
remove_field => ["message"]
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["10.237.79.147:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}
- 启动logstash
[root@m5315392269-5 logstash-7.5.1]# ./bin/logstash -f logstash.conf
Logstash配置文件介绍
- 设置输入
- 设置过滤规则
add_field:在事件中加入一个filed
tags:添加一个任意的数字作为当前事件的标签,这个标签对我们后续的工作会有帮助
参考教程链接:关于logstash导入es库的配置文件的一些理解
mutate过滤器:
mutate插件用于集成至Logstash上过滤日志数据,即在数据写入ES前进行数据清洗和数据格式化。
split函数(mutate中常用函数):将包含特定分隔符的字符串拆分成数组
参考教程链接:ELK 系列六、logstash 的mutate过滤器功能介绍 - 设置输出
遇到问题
(1)LogStash 错误:Logstash could not be started because there is already another instance usin
Sending Logstash logs to /usr/local/logstash/logstash-6.5.0/logs which is now configured via log4j2.properties
[2018-11-20T12:23:45,931][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-11-20T12:23:46,088][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2018-11-20T12:23:46,130][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
原因及解决办法
之前运行的instance有缓冲,保存在path.data里面有.lock文件(path默认指logstash解压后的目录),删除掉即可。
data$ ls -alh
总用量 20K
drwxr-sr-x 4 tingshuo staff 4.0K 11月 20 11:42 .
drwxr-sr-x 14 tingshuo staff 4.0K 11月 20 11:42 ..
drwxr-sr-x 2 tingshuo staff 4.0K 11月 20 11:42 dead_letter_queue
-rw-r--r-- 1 tingshuo staff 0 11月 20 11:42 .lock
drwxr-sr-x 2 tingshuo staff 4.0K 11月 20 11:42 queue
-rw-r--r-- 1 tingshuo staff 36 11月 20 11:42 uuid
#删除并重新启动
rm -rf .lock