6.2.3破解:https://www.jianshu.com/p/6acfeabb44f8
主要方法:更改LicenseVerifier.class和 XPackBuild.class(6.0版本不需要)
- 修改java证书代码 ---- 在Windows本地上操作
Windows需要工具:luyten-0.5.0(java代码编辑器)
/opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar
将x-pack-core-6.2.3.jar包下载到Windows本地,用luyten打开,导出LicenseVerifier.class和 XPackBuild.class分别更改代码
- LicenseVerifier.class
org.elasticresearch.license----》LicenseVerifier.class : 将此文件另外到桌面(另外后文件名自动为LicenseVerifier.java)
用notepad 编辑LicenseVerifier.java,内容如下:
package org.elasticsearch.license;
import java.nio.*;
import java.util.*;
import java.security.*;
import org.elasticsearch.common.xcontent.*;
import org.apache.lucene.util.*;
import org.elasticsearch.common.io.*;
import java.io.*;
public class LicenseVerifier
{
public static boolean verifyLicense(final License license, final byte[] encryptedPublicKeyData) {
return true;
}
public static boolean verifyLicense(final License license) {
return true;
}
}
- org.elasticsearch.xpack.core----》XPackBuild.class: 将此文件另外到桌面(另外后文件名自动为XPackBuild.java)
用notepad 编辑XPackBuild.java,内容如下:
package org.elasticsearch.xpack.core;
import org.elasticsearch.common.io.*;
import java.net.*;
import org.elasticsearch.common.*;
import java.nio.file.*;
import java.io.*;
import java.util.jar.*;
public class XPackBuild
{
public static final XPackBuild CURRENT;
private String shortHash;
private String date;
@SuppressForbidden(reason = "looks up path of xpack.jar directly")
static Path getElasticsearchCodebase() {
final URL url = XPackBuild.class.getProtectionDomain().getCodeSource().getLocation();
try {
return PathUtils.get(url.toURI());
}
catch (URISyntaxException bogus) {
throw new RuntimeException(bogus);
}
}
XPackBuild(final String shortHash, final String date) {
this.shortHash = shortHash;
this.date = date;
}
public String shortHash() {
return this.shortHash;
}
public String date() {
return this.date;
}
static {
final Path path = getElasticsearchCodebase();
String shortHash = null;
String date = null;
Label_0157: {
shortHash = "Unknown";
date = "Unknown";
}
CURRENT = new XPackBuild(shortHash, date);
}
}
- 更新java证书代码 ----- 在elk服务器上操作
将刚才在Windows本地修改的LicenseVerifier.java和XPackBuild.java上传到服务器/opt/src/bianyi,然后进行编译,并重新打包jar包
- 编译LicenseVerifier.java和XPackBuild.java
[swadmin@localhost ~]$ cd /opt/src/bianyi/ && ls
LicenseVerifier.java XPackBuild.java
[swadmin@localhost bianyi]$ javac -cp "/opt/apps/elk/elasticsearch/lib/elasticsearch-6.2.3.jar:/opt/apps/elk/elasticsearch/lib/lucene-core-7.2.1.jar:/opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar" LicenseVerifier.java
[swadmin@localhost bianyi]$ javac -cp "/opt/apps/elk/elasticsearch/lib/elasticsearch-core-6.2.3.jar:/opt/apps/elk/elasticsearch/lib/lucene-core-7.2.1.jar:/opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar" XPackBuild.jav
- 重新打包jar包
[swadmin@localhost bianyi]$ mkdir /opt/src/test/ && cd /opt/src/test/
[swadmin@localhost test]$ cp /opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/x-pack-core-6.2.3.jar /opt/src/test/
[swadmin@localhost test]$ jar -xvf x-pack-core-6.2.3.jar && rm x-pack-core-6.2.3.jar
[swadmin@localhost test]$ cp /opt/src/bianyi/LicenseVerifier.class org/elasticsearch/license/
[swadmin@localhost test]$ cp /opt/src/bianyi/XPackBuild.class org/elasticsearch/xpack/core/
[swadmin@localhost test]$ jar -cvf x-pack-core-6.2.3.jar ./*
- 将重新打包的jar包替换之前的旧x-pack-core-6.2.3.jar包
[swadmin@localhost test]$ cp x-pack-core-6.2.3.jar /opt/apps/elk/elasticsearch/plugins/x-pack/x-pack-core/
- 重新认证
- 申请一个免费license
下载链接:https://license.elastic.co/registration/download/96d3e8ef-49e7-4e6b-930a-47f695440def
将下载好的文件max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json传至elk服务器/opt/src/bianyi
- 修改license文件
[swadmin@localhost bianyi]$ cd /opt/src/bianyi/ && vim max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json
"type":"platinum"
"expiry_date_in_millis":2524579200999
"max_nodes":1000
- 禁用x-pack 配置license有效期
更新License文件,因为elasticsearch 6.0.0,现在更新license文件时,要么配置SSL\TLS,要么就禁用security。
更新之前先配置elasticsearch.yml,加入:
[swadmin@localhost ~]$ vim /opt/apps/elk/elasticsearch/config/elasticsearch.yml
xpack.security.enabled: false
- 启动elasticsearch,更新认证
即更新前面编辑的max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json认证文件)
[swadmin@localhost ~]$ elasticsearch -d
[swadmin@localhost test]$ cd /opt/src/bianyi/ && ll
max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json
[swadmin@localhost bianyi]$ curl -XPUT -u elastic 'http://127.0.0.1:9200/_xpack/license?acknowledge=true' -H "Content-Type: application/json" -d @max-mac-e8e90718-9333-49b6-817a-9098c35c0920-v5.json
出现acknowledge字符就代表成功
- 重启elasticsearch,检查更新
[swadmin@MyCloudServer ~]$ curl -XGET -u elastic "http://127.0.0.1:9200/_license"
{
"license" : {
"status" : "active",
"uid" : "e8e90718-9333-49b6-817a-9098c35c0920",
"type" : "platinum",
"issue_date" : "2018-02-24T00:00:00.000Z",
"issue_date_in_millis" : 1519430400000,
"expiry_date" : "2049-12-31T16:00:00.999Z",
"expiry_date_in_millis" : 2524579200999,
"max_nodes" : 1000,
"issued_to" : "max mac (SOLAIRE)",
"issuer" : "Web Form",
"start_date_in_millis" : 1519430400000
}
}
- Elk优化
优化指南:
删除时间范围的数据:https://juejin.im/post/58e5de06ac502e006c254145
ELK平台性能优化 http://www.th7.cn/db/nosql/201708/250381.shtml
http://blog.csdn.net/jiao_fuyou/article/details/49783861
Logstash优化 http://blog.csdn.net/ypc123ypc/article/details/78033142 https://yq.aliyun.com/articles/413002
elasticsearch优化 https://www.jianshu.com/p/29ffce0850af
安装pv (pipe views) http://blog.csdn.net/u011478909/article/details/52584935
优化: https://zhuanlan.zhihu.com/p/30888923 有具体参考方法
- elasticsearch优化
- 索引的curl管理
- 查看所有indices
[swadmin@MyCloudServer ~]$ curl -XGET http://127.0.0.1:9200/_cat/indices -u elastic
- 删除指定indices
[swadmin@MyCloudServer ~]$ curl -XDELETE "http://127.0.0.1:9200/hbet_tomcat_9002" -u elastic
备注:
获取7天前的日期:[swadmin@MyCloudServer ~]$ date +%Y.%m.%d --date="-7 day"
Curator ---删除indices https://zhuanlan.zhihu.com/p/30888923
删除索引脚本
#!/bin/bash
valite_date=$(date +%Y.%m.%d --date="-1 day")
elastic_ip=192.168.175.241
elastic_port=9200
elastic_user=elastic
elastic_pass=Passw0rd\!\*\*yibo
function delete_index(){
expect << EOF
set timeout 2
spawn curl -XDELETE http://$elastic_ip:$elastic_port/$1 -u $elastic_user ;
expect {
"elastic\'\:" { send "$elastic_pass\r" }
}
expect eof
EOF
}
##### 删除指定索引
delete_index gameplat-$valite_date
delete_index tomcat-$valite_date
delete_index logstash-nginx-$valite_date
- 查看threadpool
curl -XGET 'http://localhost:9200/_nodes/stats?pretty' -u elastic
- 查看集群信息
curl 'http://127.0.0.1:9200/_cluster/health?pretty' -u elastic
- 定期删除索引数据
参考: https://juejin.im/post/58e5de06ac502e006c254145
- elasticsearch集群搭建
参考资料:
https://www.felayman.com/articles/2017/12/12/1513085668561.html
http://cwiki.apachecn.org/pages/viewpage.action?pageId=4882617
https://www.zybuluo.com/tinadu/note/516453
节点1 --- master
[swadmin@MyCloudServer ~]$ cd /opt/apps/elk/elasticsearch/
[swadmin@MyCloudServer ~]$ cp config/elasticsearch.yml config/elasticsearch.yml.bak
[swadmin@MyCloudServer elasticsearch]$ grep -v \# config/elasticsearch.yml
cluster.name: my-application
node.name: node-1
path.data: /opt/apps/elk/elasticsearch/data/
path.logs: /opt/apps/elk/elasticsearch/logs/
network.host: 103.68.110.227
http.port: 9200
transport.tcp.port: 9600
node.master: true
discovery.zen.ping.unicast.hosts: ["103.68.110.227:9600", "103.68.110.242:9601"]
discovery.zen.minimum_master_nodes: 1
生成x-pack证书
103.68.110.223,103.68.110.225,103.68.110.227,103.68.110.242
参考资料:https://segmentfault.com/a/1190000012789290
[swadmin@MyCloudServer es_crt]$ cd /opt/apps/elk/elasticsearch/config/
[swadmin@MyCloudServer elk]$ /opt/apps/elk/elasticsearch/bin/x-pack/certgen
依次输入:my_cluster.zip ---> my_cluster ---> my_cluster ---> 103.68.110.242 ---> enter ---> enter
[swadmin@MyCloudServer config]$ unzip my_cluster.zip
[swadmin@MyCloudServer ~]$ vim /opt/apps/elk/elasticsearch/config/elasticsearch.yml ##末行追加
################x-pack相关配置
######停用x-pack
##xpack.security.enabled: false
######所有节点进行如下配置,证书秘钥路径一定要正确
xpack.ssl.key: my_cluster/my_cluster.key
xpack.ssl.certificate: my_cluster/my_cluster.crt
xpack.ssl.certificate_authorities: ca/ca.crt
xpack.security.transport.ssl.enabled: true
[swadmin@es_01 config]$ ../bin/x-pack/setup-passwords interactive ###重新配置x-pack密码
节点2
[swadmin@MyCloudServer ~]$ cd /opt/apps/elk/elasticsearch/
[swadmin@MyCloudServer ~]$ cp config/elasticsearch.yml config/elasticsearch.yml.bak
[swadmin@MyCloudServer elasticsearch]$ grep -v \# config/elasticsearch.yml
cluster.name: my-application
node.name: node-2
path.data: /opt/apps/elk/elasticsearch/data/
path.logs: /opt/apps/elk/elasticsearch/logs/
network.host: 103.68.110.242
http.port: 9201
transport.tcp.port: 9601
discovery.zen.ping.unicast.hosts: ["103.68.110.227:9600", "103.68.110.242:9601"]
discovery.zen.minimum_master_nodes: 1
从节点1将my_cluster 和ca目录拉过来放到/opt/apps/elk/elasticsearch/config/下
- 其他优化
- 日志优化
[swadmin@MyCloudServer]$ vim log4j2.properties
logger.index_search_slowlog_rolling.level = info ##默认为trace
- 启动多个elasticsearch
https://my.oschina.net/u/3470972/blog/1586637
http://knktc.com/2016/06/10/elasticsearch-multiple-instances/
elasticsearch -Epath.conf=/opt/apps/elk/elasticsearch/config/my_cluster/ -Ecluster.name=my_cluster -Enode.name=node_2
- 开放elasticsearch给外网
[swadmin@MyCloudServer ~]$ vim /opt/apps/elk/elasticsearch/config/elasticsearch.yml
transport.host: localhost
network.host: 0.0.0.0
[swadmin@MyCloudServer]$ vim /opt/apps/elk/elasticsearch/config/jvm.options
-Xms5g
-Xmx5g
备注:内存分配物理内存1/2或1/4
[swadmin@MyCloudServer]$ vim /opt/apps/elk/elasticsearch/bin/elasticsearch
ES_JAVA_OPTS="-Xms6g -Xmx6g"
完整优化
https://www.cnblogs.com/ningskyer/articles/5788667.html
- 禁用x-pack功能 ---有必要的时候
[swadmin@vhost-elk ~]$ vim /opt/apps/elk/elasticsearch-01/config/elasticsearch.yml
xpack.security.enabled: false
- Logstash优化
- 利用Pipeline启动多个管道
参考:
http://blog.csdn.net/ypc123ypc/article/details/78033142
http://blog.csdn.net/ypc123ypc/article/details/69945031
http://blog.51niux.com/?id=205
[swadmin@MyCloudServer]$ vim /opt/apps/elk/logstash/config/pipelines.yml
- pipeline.id: tomcat_log
queue.type: persisted
path.config: "/opt/apps/elk/logstash/conf.d/elk_tomcat.conf"
pipeline.workers: 6
pipeline.batch.size: 1000
pipeline.batch.delay: 5
- pipeline.id: nginx_log
path.config: "/opt/apps/elk/logstash/conf.d/elk_nginx.conf"
pipeline.workers: 4
pipeline.batch.size: 800
pipeline.batch.delay: 5
- pipeline.id: gameplat _log
queue.type: persisted
path.config: "/opt/apps/elk/logstash/conf.d/elk_gameplat.conf"
pipeline.batch.size: 1000
pipeline.batch.delay: 5
- 内存和线程优化
[swadmin@MyCloudServer]$ vim /opt/apps/elk/logstash/config/jvm.options
-Xms6g
-Xmx6g
Logstash参数表
参数 |
类别 |
说明 |
|
1 |
LS_HEAP_SIZE |
LS |
Logstash堆内存大小,默认1g |
2 |
-w |
LS启动 |
logstash线程数,默认与cpu数相同 |
3 |
-b |
LS启动 |
Batch数,即logstash取多少数据进行一次filter,默认125 |
4 |
redis.threads |
LS input |
Redis线程数,默认1 |
5 |
redis.batch_count |
LS input |
Redis每次pop的数量,默认1 |
6 |
es.workers |
LS output |
Es提交线程,默认1 |
7 |
es.flush_size |
LS output |
ESbulk提 |
8 |
-l |
LS启动 |
产生日志文件 |
备注:
带pv启动logstash
logstash -f logstash_dots_zzm1.conf -l ./logstash_zzm1.log -b 8000 | pv -abt >/dev/null
#####
再次优化索引
用一个
If ([fields][source] =~ "gameplat")
- x-pack常见参数优化
参考资料:https://www.felayman.com/articles/2017/12/12/1513085668561.html
- elk安全告警机制
参考:
https://xizhibei.github.io/2017/11/19/alerting-with-elastalert/
https://github.com/xuyaoqiang/elastalert-dingtalk-plugin 钉钉报警插件
钉钉添加群---》群设置----》群机器人----》添加机器人---》编辑
sudo pip install setuptools --upgrade
- GeoIP地图库
Logstash下载GeoData
[swadmin@MyCloudServer]$ cd /opt/apps/elk/logstash/ && mkdir geoData/
[swadmin@MyCloudServer logstash]$ cd geoData && wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
[swadmin@logstash logstash]$ vim /opt/apps/elk/logstash/conf.d/elk_nginx.conf
.....
geoip {
source => "clientip"
target => "geoip"
database => "/opt/apps/elk/logstash/geoData/GeoLite2-City_20180306/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
....
index => "logstash-nginx-%{+YYYY.MM.dd}"
...
备注:index必须以logstash开头
- 常见报错
- 删掉tags的_grokparsefailure
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
tag_on_failure => []
}