测试环境es有warn:
[o.e.g.DanglingIndicesState] [elk-app10] failed to list dangling indices
java.nio.file.FileSystemException: /data/elasticsearch/data/nodes/0/indices: Too many open files in system
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427) ~[?:?]
at java.nio.file.Files.newDirectoryStream(Files.java:457) ~[?:?]
at org.elasticsearch.env.NodeEnvironment.availableIndexFoldersForPath(NodeEnvironment.java:901) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.env.NodeEnvironment.availableIndexFolders(NodeEnvironment.java:867) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.gateway.MetaStateService.loadIndicesStates(MetaStateService.java:89) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.gateway.DanglingIndicesState.findNewDanglingIndices(DanglingIndicesState.java:137) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.gateway.DanglingIndicesState.findNewAndAddDanglingIndices(DanglingIndicesState.java:122) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.gateway.DanglingIndicesState.processDanglingIndices(DanglingIndicesState.java:87) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.gateway.DanglingIndicesState.clusterChanged(DanglingIndicesState.java:191) ~[elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:495) ~[elasticsearch-6.7.1.jar:6.7.1]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) [?:?]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) [?:?]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) [?:?]
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:492) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:475) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-6.7.1.jar:6.7.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-6.7.1.jar:6.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:?]
at java.lang.Thread.run(Thread.java:745) [?:?]
这是因为系统打开的文件太多了,可以用ulimit -a查看当前系统文件最大打开数
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 191525
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 512000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 512000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
栏open files不是默认的1024,没毛病。
lsof |wc -l
可以看已经打开文件数量。8万了。
file-max是设置 系统所有进程一共可以打开的文件数量 。
再看看:cat /proc/sys/fs/file-max
65536
这个小了。还需要改。
echo 52000 > /proc/sys/fs/file-max
再来看看,cat /proc/sys/fs/file-max
52000
通常线上环境不会出现这种问题,运维都会提前配好了。