环境准备:
操作系统:CentOS Linux release 7.4.1708 (Core) 虚拟机即可
兼容矩阵
最近软件更新换代很快啊,到底搭建哪一款ELK适合呢。来官网看下兼容矩阵:
根据兼容矩阵,选择一款适合自己的,由于小编操作系统centos自带了openjdk1.8.因此还需要考虑到是否兼容java1.8的环境。
从兼容矩阵看到,ES8已经放弃了JDK1.8。因此小编选择了ES7.10来作为实验品。
软件列表如下:
elasticsearch-7.10.1-linux-x86_64.tar.gz
kibana-7.10.0-linux-x86_64.tar.gz
logstash-7.10.0-linux-x86_64.tar.gz
就选择这3个来搭建ELK环境。
下载地址
从开源社区下载:下载中心 - Elastic 中文社区
ElasticSearch安装
先安装elastic search,赋权给非root用户。我喜欢visudo。简单快捷方便。
ES 有点注意的是,它不能用root用户启动。所以提前建好非root用户。
如下命令暂时记下,需要再ES启动报错执行。偷懒了,记得是需要改些系统配置的。
操作命令
301 2022-03-23 11:37:28 ulimit -S -n
302 2022-03-23 11:37:31 ls /etc/security/limits.conf
303 2022-03-23 11:37:34 vim /etc/security/limits.conf
304 2022-03-23 11:39:14 echo "fs.file-max = 1000000">> /etc/sysctl.conf
305 2022-03-23 11:39:14 echo "net.core.somaxconn = 32768">> /etc/sysctl.conf
306 2022-03-23 11:39:14 echo "net.ipv4.tcp_tw_recycle = 0">> /etc/sysctl.conf
307 2022-03-23 11:39:14 echo "net.ipv4.tcp_syncookies = 0">> /etc/sysctl.conf
308 2022-03-23 11:39:14 echo "vm.overcommit_memory = 1">> /etc/sysctl.conf
309 2022-03-23 11:39:15 sysctl -p
310 2022-03-23 11:39:39 cat >> /etc/systemd/system/cpupower.service << EOF
311 2022-03-26 16:41:00 [Unit]
312 2022-03-26 16:41:00 Description=CPU performance
313 2022-03-26 16:41:00 [Service]
314 2022-03-26 16:41:00 Type=oneshot
315 2022-03-26 16:41:00 ExecStart=/usr/bin/cpupower frequency-set --governor performance
316 2022-03-26 16:41:00 [Install]
317 2022-03-26 16:41:00 WantedBy=multi-user.target
318 2022-03-26 16:41:00 EOF
319 2022-03-23 11:39:44 systemctl daemon-reload
320 2022-03-23 11:39:48 systemctl enable cpupower.service
321 2022-03-23 11:39:54 systemctl start cpupower.service
322 2022-03-23 11:40:14 echo never > /sys/kernel/mm/transparent_hugepage/enabled
323 2022-03-23 11:40:15 echo never > /sys/kernel/mm/transparent_hugepage/defrag
324 2022-03-23 11:40:23 vi /etc/udev/rules.d/60-tidb-schedulers.rules
325 2022-03-23 11:40:42 udevadm control --reload-rules
326 2022-03-23 11:40:43 udevadm trigger --type=devices --action=change
327 2022-03-23 11:40:46 cat >> /etc/systemd/system/cpupower.service << EOF
328 2022-03-26 16:41:00 [Unit]
329 2022-03-26 16:41:00 Description=CPU performance
330 2022-03-26 16:41:00 [Service]
331 2022-03-26 16:41:00 Type=oneshot
332 2022-03-26 16:41:00 ExecStart=/usr/bin/cpupower frequency-set --governor performance
333 2022-03-26 16:41:00 [Install]
334 2022-03-26 16:41:00 WantedBy=multi-user.target
335 2022-03-26 16:41:00 EOF
336 2022-03-23 11:40:50 systemctl daemon-reload
337 2022-03-23 11:40:50 systemctl enable cpupower.service
338 2022-03-23 11:40:51 systemctl start cpupower.service
339 2022-03-23 11:40:55 cat /sys/kernel/mm/transparent_hugepage/enabled
340 2022-03-23 11:40:59 cat /sys/block/sd[bc]/queue/scheduler
341 2022-03-23 11:41:24 vim /etc/security/limits.conf
ES配置
ES配置文件更改:
config文件夹下的jvm.options更改。
-Xms10g
-Xmx10g
来它10个G的内存,否则会很慢。
config文件夹下的elasticsearch.yml,有效配置为:
cluster.name: my-application
node.name: node-1
path.logs: /data/path/to/logs
path.logs: /data/path/to/logs
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node-1"]
其他配置暂且搁置,不影响Es使用。启动即可使用。
保证高可用
为了保证一个高可用,用脚本实现下单机版本的高可用
#!/bin/bash
nnn1=$(ps -ef | grep elasticsearch | grep -v grep | grep -v monitor | wc -l)
echo $nnn1
for i in $(ps -ef | grep elasticsearch | grep -v grep | grep -v monitor |awk '{print $2}');do echo $i;done;
ps -ef | grep elasticsearch | grep -v grep | grep -v monitor
if [ $nnn1 -ne 2 ]; then
echo 'wrong,pease restart the elasticsearch'
for i in $(ps -ef | grep elasticsearch | grep -v grep |grep -v monitor |awk '{print $2}');do kill -9 $i;done;
echo "restart now ...."
/data/software/elasticsearch/bin/elasticsearch -d
echo "restart over ....."
fi
在系统定时器里加上
[sysadm@tidb196 config]$ crontab -l
*/2 * * * * sh /data/software/elasticsearch/bin/monitor.sh &
只要ES有变动,两分钟监控一次,然后启动,可以保证ES不会挂掉吧。。。。
哈哈。这样,只要发现es服务器挂,就启动。优点low。
kibana安装
这个更简单了,直接把压缩包解压即可。
有效配置
server.port: 5601
server.host: "0.0.0.0"
server.name: "your-hostname"
elasticsearch.requestTimeout: 60000
i18n.locale: "zh-CN"
然后sbin目录下kiban运行即可。
也用脚本和监控来了下。
#!/bin/bash
kn=$(ss -antlup | grep 5601 | wc -l)
echo $kn
for i in $(ss -antlup | grep 5601 | awk '{print $7}' | awk -F"," '{print $2}' | awk -F"=" '{print $2}');do echo $i;done;
if [ $kn -ne 2 ]; then
echo 'wrong,pease restart the kibana'
for i in $(ss -antlup | grep 5601 | awk '{print $7}' | awk -F"," '{print $2}' | awk -F"=" '{print $2}');do kill -9 $i;done;
echo "kibana restart now ...."
/data/software/kibana/bin/kibana &
echo "kibana restart over ....."
fi
[sysadm@tidb196 bin]$ crontab -l
*/2 * * * * sh /data/software/elasticsearch/bin/monitor.sh &
*/2 * * * * sh /data/software/kibana/bin/monitor.sh &
*/2 * * * * sh /data/software/logstash/bin/monitor.sh &
logstash安装和配置
input {
gelf {
port => 1560
use_tcp => true
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["http://11.10.32.196:9200"]
index => "logstash-%{service_name}-%{+YYYY-MM-dd}"
}
}
#!/bin/bash
ln=$(ps -ef | grep logstash | grep -v grep | grep -v monitor | wc -l)
echo $ln
if [ $ln -ne 1 ]; then
echo 'wrong,pease restart the logstash'
for i in $(ps -ef | grep logstash | grep -v grep |grep -v monitor |awk '{print $2}');do kill -9 $i;done;
echo "logstash restart now ...."
/data/software/logstash/bin/logstash -f /data/software/logstash/config/logstash.conf --config.reload.automatic &
echo "logstash restart over ....."
fi