编译模板文件bigdata.template:
{ "template": "bigdata-template", "settings": { "index.number_of_shards": 5, "number_of_replicas": 1 }, "mappings": { "_default_": { "_all": { "enabled": true, "omit_norms": true }, "dynamic_templates": [{ "message_field": { "match": "message", "match_mapping_type": "string", "mapping": { "type": "string", "index": "analyzed", "omit_norms": true, "fielddata": { "format": "disabled" } } } }, { "string_fields": { "match": "*", "match_mapping_type": "string", "mapping": { "type": "string", "index": "not_analyzed", "doc_values": true } } }], "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "string", "index": "not_analyzed" } } } } }
Logstash配置文件 nginx.conf:
input { file { path => "/usr/local/openresty/nginx/logs/user2.log" type => "nginx-bigdata" codec => "json" } } filter { json { source => "u_data" } } output { if [type] == "nginx-bigdata" { elasticsearch { hosts => ["172.17.213.60:9200", "172.17.213.61:9200"] index => "nginx-bigdata" manage_template => false template_overwrite => true template_name => "bigdata-template" template => "/usr/local/logstash-6.2.4/bigdata.template" document_type => "nginx-bigdata" } } }
Nginx的配置文件中关于日志格式的配置:
此处我只保留了需要的一个字段范围
log_format userlog escape=json '{"u_data":"$u_data"}';
access_log logs/user.log userlog;
产生的日志格式:
{"u_data":"{\"appid\":\"nchaopai\",\"args\":{\"contentId\":0,\"duration\":111811,\"parentId\":0,\"totaltime\":0,\"type\":0},\"bk\":\"-\",\"cp_ver\":\"3.0.5\",\"duid\":\"2cba98f8ddc18464\",\"e\":\"nchaopai.main.stay-duration\",\"os\":\"A\",\"ts\":1572584611,\"ver\":\"8.11.11\"}"}
之后在Kibana里看到就是这样的: