一.前提条件
1.操作系统版本:
Centos 6.5
2.安装顺序
a. elasticsearch
b.kibana
c.logstash
d.beats
3.安装jdk-8u131-linux-x64.tar.gz
#vi /etc/profile export JAVA_HOME=/usr/local/java export CLASSPATH=$JAVA_HOME/lib/ export PATH=$PATH:$JAVA_HOME/bin #source /etc/profile
4.ES stack基本概念
database → tables → rows → column indices → type → document → fields
二.安装ES
1.创建es用户启动ES,否则会报错
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.3.jar:6.2.3]
#useradd es #passwd es #chown -R es:es /usr/local/elasticsearch-6.2.3 #/usr/local/elasticsearch-6.2.3/bin/elasticsearch -d #后台启动ES
修改es配置文件内容:
[es@elk config]$ cat elasticsearch.yml | grep -v '^#' path.data: /usr/local/elasticsearch-6.2.3/data path.logs: /usr/local/elasticsearch-6.2.3/logs network.host: 172.16.68.150 http.port: 9200 [es@elk config]$
如启动失败,通过日志分析并处理:
[4] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
#vi /etc/security/limits.conf * soft nofile 65536 * hard nofile 131072 * soft nproc 2048 * hard nproc 4096
[2]: max number of threads [1024] for user [es] is too low, increase to at least [4096]
#vi /etc/security/limits.d/90-nproc.conf 修改为* soft nproc 4096
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
#vi /etc/sysctl.conf 添加 vm.max_map_count = 262144
[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
在elasticsearch.yml中配置bootstrap.system_call_filter为false,注意要在Memory下面: bootstrap.memory_lock: false bootstrap.system_call_filter: false
三.安装Kibana
#tar -zxf kibana-6.2.3-linux-x86_64.tar.gz #mv /tmp/kibana-6.2.3-linux-x86_64 /usr/local/kibana # cat kibana.yml | grep -v "^#" server.port: 5601 server.host: "172.16.68.150" elasticsearch.url: "http://172.16.68.150:9200" kibana.index: ".kibana"
默认情况下,kibana进程处于前台运行,需要将其设置为后台运行
#nohup /usr/local/kibana/bin/kibana & #jobs [1]+ Running nohup /usr/local/kibana/bin/kibana & # netstat -antlp | grep :5601 tcp 0 0 172.16.68.150:5601 0.0.0.0:* LISTEN 1977/node
检查Kibaba运行状态:
http://IP:5601/status
四.安装logstash
#tar -zxf /tmp/logstash-6.2.3.tar.gz #mv /tmp/logstash-6.2.3 /usr/local/logstash 编辑配置文件 #vi /usr/local/logstash.conf input { beats { port => 5044 } } output { elasticsearch { hosts => "172.16.68.150:9200" } }
#测试配置文件语法正确性
#bin/logstash -f logstash.conf --config.test_and_exit
#后台运行Logstash服务
#nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/logstash.conf --config.reload.automatic & #The "--config.reload.automatic" option enables automatic config reloading so that you don’t have to stop and restart Logstash every time you modify the configuration file.
五.安装filebeat插件
1.在需要收集日志的客户端服务器中执行
#rpm -ivh filebeat-6.2.3-x86_64.rpm #vi /etc/filebeat/filebeat.yml 修改具体要获取的日志绝对路径以及输出的logstash ip/port ———————————————————————————————————— [root@webtest filebeat]# cat /etc/filebeat/filebeat.yml | grep -v "#" | grep -v "^$" filebeat.prospectors: - type: log enabled: true paths: - /data/tomcatwww/logs/b2c/root.log fields: service: 68_224_b2c_root scan.order: desc - type: log enabled: true paths: - /data/tomcatwww/logs/b2c/order.log fields: service: 68_224_b2c_order scan.order: desc - type: log enabled: true paths: - /data/tomcatwww/logs/b2c/sale.log fields: service: 68_224_b2c_sale scan.order: desc - type: log enabled: true paths: - /data/tomcatwww/logs/b2c/srv.log fields: service: 68_224_b2c_srv scan.order: desc filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 output.logstash: hosts: ["172.16.68.150:5044"] ———————————————————————————————————— #启动filebeat #/etc/init.d/filebeat start #ps -ef | grep filebeat #ps -ef | awk '{print $1}'
六. 安装X-Pack(收费)
1.安装x-pack在es中
[es@elk bin]$ ./elasticsearch-plugin install file:///tmp/x-pack-6.2.3.zip -> Downloading file:///tmp/x-pack-6.2.3.zip [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.io.FilePermission \\.\pipe\* read,write * java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries * java.lang.RuntimePermission getClassLoader * java.lang.RuntimePermission setContextClassLoader * java.lang.RuntimePermission setFactory * java.net.SocketPermission * connect,accept,resolve * java.security.SecurityPermission createPolicy.JavaPolicy * java.security.SecurityPermission getPolicy * java.security.SecurityPermission putProviderProperty.BC * java.security.SecurityPermission setPolicy * java.util.PropertyPermission * read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin forks a native controller @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ This plugin launches a native controller that is not subject to the Java security manager nor to system call filters. Continue with installation? [y/N]y Elasticsearch keystore is required by plugin [x-pack-security], creating... -> Installed x-pack with: x-pack-deprecation,x-pack-monitoring,x-pack-upgrade,x-pack-ml,x-pack-logstash,x-pack-watcher,x-pack-security,x-pack-graph,x-pack-core [es@elk bin]$ ll
2.设置默认用户的密码
[es@elk x-pack]$ ./setup-passwords interactive Initiating the setup of passwords for reserved users elastic,kibana,logstash_system. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [elastic]
3.安装x-pack在kibana中
#./elasticsearch-plugin install file:///tmp/x-pack-6.2.3.zip [root@elk bin]# ./kibana-plugin install file:///tmp/x-pack-6.2.3.zip Attempting to transfer from file:///tmp/x-pack-6.2.3.zip Transferring 270035965 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Optimizing and caching browser bundles... Plugin installation complete [root@elk bin]# [root@elk bin]# [root@elk bin]# ./kibana-plugin list [email protected]
4.安装x-pack在logstash中
#./logstash-plugin install file:///tmp/x-pack-6.2.3.zip