本篇文章继续给大家介绍ELFK日志分析系统,包括DSL语句、IK分词器、索引模板和Kibana可视化工具的部署与使用。在介绍之前我们先创建索引,写入一些数据。
创建索引
PUT http://10.0.0.101:19200/koten-shopping
{
"mappings" :{
"properties": {
"title": {
"type": "text"
},
"price": {
"type": "float"
},
"brand": {
"type": "keyword"
},
"item": {
"type": "text"
},
"group": {
"type": "byte"
},
"ip_addr" : {
"type": "ip"
},
"author": {
"type": "text"
}
}
},
"settings": {
"number_of_shards":3,
"number_of_replicas":0
}
}
批量写入10条测试数据
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"BMW1系运动轿车 125i M运动曜夜版","price":249900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=1&rangeCode=F52&packageCode=2W71_0ZMB&sop=1672502400000","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"BMW2系四门轿跑车 225i M运动曜夜套装","price":298900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=2&rangeCode=F44&packageCode=31AK_0ZSM&sop=1667232000000","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"新BMW3系 330i M运动曜夜套装","price":381900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=3&rangeCode=G20&packageCode=24FF_0ZMC&sop=1672502400000","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"THE i3 eDrive40L曜夜运动套装","price":413900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=3&rangeCode=G28BEV&packageCode=81BE_0ZCP&sop=1672502400000","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"BMW4系双门轿跑车 430i M运动曜夜套装","price":479900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=4&rangeCode=G22&packageCode=51AP_0ZSM&sop=1677600000000","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"BMW4系敞篷轿跑车 430i M运动曜夜套装","price":576900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=4&rangeCode=G23&packageCode=11AT_0ZSM","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"纯电动BMW i4 eDrive40","price":469900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=4&rangeCode=G26BEV&packageCode=71AW_0ZOM&sop=1677600000000","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"BMW5系Li 530Li xDrive M运动套装","price":495500,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=5&rangeCode=G38&packageCode=21AF_0ZLU","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"BMW6系GT 630i M运动大旅行家版","price":699900,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=6&rangeCode=G32&packageCode=21BP_0ZLU","group":3,"auther":"koten"}
{ "create": { "_index": "koten-shopping"} }
{ "ip_addr": "221.194.154.181" ,"title":"新BMW7系 740Li 尊享型 豪华套装","price":1269000,"brand":"宝马","item":"https://www.bmw-emall.cn/newcar/design?series=7&rangeCode=G70&packageCode=11EH_0ZOM&sop=1677600000000","group":3,"auther":"koten"}
DSL语句
1、match模糊查询,若字段brand是包含宝马的也可以查询到
由于brand字段类型是keyword,所以match搜索也不会将宝马两个字分割开来匹配,也算是精准匹配。如果是text字段类型就会一个一个的进行匹配了。
GET http://10.0.0.101:19200/oldboyedu-shopping/_search
{
"query":{
"match":{
"brand":"宝马"
}
}
}
2、match_phrase精准查询,只能查询到brand字段是宝马的,多一个字少一个字都匹配不到,由于我们写入了数据,不能修改字段类型了,不能将text修改为keyword,所以我们采取精准查询方式
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query":{
"match_phrase":{
"brand":"宝马"
}
}
}
3、match_all全量查询,一般用不到,默认不写参数就是全量
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query":{
"match_all":{}
}
}
4、分页查询,size表示每页显示几条数据,from代表跳过数据的条数,查看第N页,form的值可以用(N-1)*size来计算,举例,一共10条数据,我们3条数据一页,想看最后一页就是10除以3等于3余1,有余数所以3+1,最后一页为第4页,(4-1)*3=9,所以from为9
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"match_phrase":{
"brand": "宝马"
}
},
"size":3,
"from": 9
}
5、查看指定字段,可以筛选出自己想看的一些字段信息
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query":{
"match_phrase":{
"author":"koten"
}
},
"_source":["title","price","author"]
}
6、排序
sort代表排序,price代表基于价格进行排序,order表示排序的方式,分别为desc表示降序,asc表示升序
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query":{
"match_phrase":{
"author":"koten"
}
},
"sort":{
"price":{
"order":"desc"
}
}
}
7、查询存在某个字段的文档
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query":{
"exists":{
"field":"author"
}
}
}
8、语法高亮
highlight表示进行语法高亮,pre_tags指定高亮的前缀标签,post_tags指定高亮的后缀标签,fields表示对哪个字段进行语法高亮,在搜索的时候,我们一般将搜索的关键词进行语法高亮
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"match": {
"title": "运动"
}
},
"sort": {
"price": {
"order": "desc"
}
},
"size":5,
"_source":["title","price","author"]
,
"highlight": {
"pre_tags": [
"<span style='color:red;'>"
],
"post_tags": [
"</span>"
],
"fields": {
"title": {}
}
}
}
9、多条件查询
必要条件must,两个条件都符合才会筛选出来
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"brand": "宝马"
}
},
{
"match": {
"price": 381900
}
}
]
}
}
}
应该满足的条件should
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"bool": {
"should": [
{
"match": {
"price": 249900
}
},
{
"match": {
"price": 381900
}
}
]
}
}
}
必须不满足的条件must-not
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"bool": {
"must_not": [
{
"match": {
"price": 381900
}
}
]
}
}
}
boll,可以匹配多个条件查询,其中must为必须匹配的条件,must_not为必须不匹配的条件,即与must相反,should,不是必要条件,满足其中的一个即可,也可以使用minimum_should_match来限制满足要求的条件数量。
10、范围查询
filter代表过滤,range代表范围查询,price基于价格进行匹配,gte代表大于等于,gt代表大于,lte代表小于等于,lt代表小于
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"brand": "宝马"
}
}
],
"filter": {
"range": {
"price": {
"gte": 400000,
"lt": 500000
}
}
}
}
}
}
11、精确匹配多个值
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"terms": {
"price": [
249900,
413900
]
}
}
}
12、权重案例
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"brand": "宝马"
}
}
],
"should": [
{
"match_phrase": {
"price": {
"query": "1269000",
"boost": 10
}
}
},
{
"match_phrase": {
"price": {
"query": "298900",
"boost": 2
}
}
}
]
}
},
"highlight": {
"fields": {
"title": {},
"brand": {}
}
}
}
13、聚合查询
统计每个品牌的数量
GET http://10.0.0.101:19200/koten-shopping/_search
{
"aggs": {
"koten_brand_group": {
"terms": {
"field": "brand"
}
}
},
"size": 0
}
统计商品中最贵的一个
GET http://10.0.0.101:19200/koten-shopping/_search
{
"aggs": {
"koten_max_shopping": {
"max": {
"field": "price"
}
}
},
"sort": {
"price": {
"order": "desc"
}
},
"_source": ["price","group","brand","title","auther"],
"size": 1
}
统计3组统计的平均商品价格
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"match": {
"group": 3
}
},
"aggs": {
"koten_avg_shopping": {
"avg": {
"field": "price"
}
}
},
"size": 0
}
统计3组的商品价格总数
GET http://10.0.0.101:19200/koten-shopping/_search
{
"query": {
"match": {
"group": 3
}
},
"aggs": {
"koten_sum_shopping": {
"sum": {
"field": "price"
}
}
},
"size": 0
}
IK分词器部署
IK分词器的作用在于在分词的时候,ES默认的分词对中文不利,只会把中文的每个字分开,而我们使用IK分词器,可以把一些常见的词语当做一个整体分开,也可以自定义一些词语。
1、内置的标准分词-分析英文
默认以空隔为分割
GET http://10.0.0.101:19200/_analyze
{
"analyzer": "standard",
"text": "My name is koten, and I'm 18 years old !"
}
2、内置的标准分词器-分析中文
会把每个字都分开
GET http://10.0.0.101:19200/_analyze
{
"analyzer": "standard",
"text": "你的博客写的真好"
}
3、安装IK分词器,集群的每个机器都要部署,滚动式部署
下载地址:https://github.com/medcl/elasticsearch-analysis-ik
切记安装版本一定要与ES的版本一致
[root@ELK101 ~]# rz -E
[root@ELK101 ~]# cd /koten/softwares/elasticsearch-7.17.5/plugins/
[root@ELK101 plugins]# install -d ik -o koten -g koten
[root@ELK101 plugins]# cd ik/
[root@ELK101 ik]# unzip ~/elasticsearch-analysis-ik-7.17.5.zip
[root@ELK101 ik]# chown -R koten:koten *
[root@ELK101 ik]# systemctl restart es7
[root@ELK101 plugins]# scp -r ik 10.0.0.102:`pwd`
[root@ELK101 plugins]# scp -r ik 10.0.0.103:`pwd`
[root@ELK102 ~]# chown -R koten:koten /koten/softwares/elasticsearch-7.17.5/plugins/ik
[root@ELK103 ~]# chown -R koten:koten /koten/softwares/elasticsearch-7.17.5/plugins/ik
[root@ELK102 ~]# systemctl restart es7
[root@ELK103 ~]# systemctl restart es7
4、测试分词器
ik_max_word表示细粒度拆分,ik_smart表示粗粒度拆分,两者区别是,细粒度拆分会将所有可能的词语拆分出来,而粗粒度拆分就是简单拆分成一份成二,一分成三
GET http://10.0.0.101:19200/_analyze
{
"analyzer": "ik_max_word",
"text": "中华人民共和国人民大会堂"
}
粗粒度拆分
GET http://10.0.0.101:19200/_analyze
{
"analyzer": "ik_smart",
"text": "中华人民共和国人民大会堂"
}
5、自定义词典
[root@ELK101 ~]# cd /koten/softwares/elasticsearch-7.17.5/plugins/ik/config/
[root@ELK101 config]# mkdi
mkdict mkdir
[root@ELK101 config]# mkdir koten-custom-dic
[root@ELK101 config]# ls
extra_main.dic main.dic
extra_single_word.dic preposition.dic
extra_single_word_full.dic quantifier.dic
extra_single_word_low_freq.dic stopword.dic
extra_stopword.dic suffix.dic
IKAnalyzer.cfg.xml surname.dic
koten-custom-dic
[root@ELK101 config]# cat koten-custom-dic/extra_custom.dic
真6
阿啦啦啦
写的
[root@ELK101 config]# cat IKAnalyzer.cfg.xml|grep koten <entry key="ext_dict">koten-custom-dic/extra-custom.dic</entry> #修改此行使配置文件生效
#再次重启服务
[root@ELK101 plugins]# scp -r ik 10.0.0.102:`pwd`
[root@ELK101 plugins]# scp -r ik 10.0.0.103:`pwd`
[root@ELK102 ~]# chown -R koten:koten /koten/softwares/elasticsearch-7.17.5/plugins/ik
[root@ELK103 ~]# chown -R koten:koten /koten/softwares/elasticsearch-7.17.5/plugins/ik
[root@ELK101 ~]# systemctl restart es7 #注意间隔开时间,滚动式重启
[root@ELK102 ~]# systemctl restart es7
[root@ELK103 ~]# systemctl restart es7
6、测试分词器
细粒度拆分
GET http://10.0.0.101:19200/_analyze
{
"analyzer": "ik_max_word",
"text": "你的博客写的真6阿啦啦啦"
}
粗粒度拆分
GET http://10.0.0.101:19200/_analyze
{
"analyzer": "ik_smart",
"text": "你的博客写的真6阿啦啦啦"
}
索引模板
索引模板是创建索引的一种方式,当数据写入指定索引时,如果该索引不存在,则根据索引名称匹配相应索引模板的话,会根据模板的配置而创建索引。索引模板仅对新创建的索引生效,对已经创建你的索引是没有作用的。
一、查看
1、查看所有的模板
GET http://10.0.0.103:19200/_template
2、查看指定的索引模板
GET http://10.0.0.103:19200/_template/.monitoring-es
二、增加
增加索引模板
POST http://10.0.0.103:19200/_template/koten
{
"aliases": {
"DBA": {},
"SRE": {},
"K8S": {}
},
"index_patterns": [
"koten*"
],
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 0
}
},
"mappings": {
"properties":{
"ip_addr": {
"type": "ip"
},
"access_time": {
"type": "date"
},
"address": {
"type" :"text"
},
"name": {
"type": "keyword"
}
}
}
}
创建索引测试
PUT http://10.0.0.103:19200/koten123
三、修改
更改其实就是再提交一次,会覆盖掉原来的模板配置,修改索引模板后,之前创建的索引并不会修改配置
POST http://10.0.0.103:19200/_template/koten
{
"aliases": {
"Nginx": {}
},
"index_patterns": [
"koten*"
],
"settings": {
"index": {
"number_of_shards": 3,
"number_of_replicas": 0
}
},
"mappings": {
"properties":{
"ip_addr": {
"type": "ip"
},
"access_time": {
"type": "date"
},
"address": {
"type" :"text"
},
"name": {
"type": "keyword"
}
}
}
}
四、删除
删除模板后,之前根据模板创建的索引不受影响
DELETE http://10.0.0.103:19200/_template/koten
Kibana安装部署与使用
一、安装部署
1、下载Kibana,下载的版本需要跟ES版本保持一致
[root@ELK101 ~]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.5-x86_64.rpm
2、安装kibana
[root@ELK101 ~]# rpm -ivh kibana-7.17.5-x86_64.rpm
3、修改kibana的配置文件
[root@ELK101 ~]# egrep -v "^#|^$" /etc/kibana/kibana.yml
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://10.0.0.101:19200","http://10.0.0.102:19200","http://10.0.0.103:19200"]
i18n.locale: "zh-CN"
4、启用kibana
[root@ELK101 ~]# systemctl start kibana
[root@ELK101 ~]# netstat -tnulp|grep 5601
tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 6631/node
5、访问kibana的webUI
http://10.0.0.101:5601,首次打开会提示添加集成,我们可以选择自己浏览
成功进入主页
此时我们的es中已经储存了kibana的索引
6、创建仪表板,添加样本数据
添加后可以看到我们的样本仪表盘,我们之后可以根据自己的需求制作类似的仪表盘数据
二、查看索引
三、管理索引
选中之后可以管理索引
创建索引模板
四、添加字段
创建好后可以在索引模式中添加字段
五、查询索引数据
如果ES中有数据需要查询,先创建索引模式
创建好后点discover
可以根据字段名称和KQL语句去查询我们的索引内容
六、开发工具
kibana相较于api,可以实现大部分的功能,但是遇到聚合求平均值之类的需求,就解决不了了,我们可以在kibana的开发工具中输入api去实现需求
我是koten,10年运维经验,持续分享运维干货,感谢大家的阅读和关注!