一、参考文档路径:
http://docs.ceph.org.cn/rados/troubleshooting/memory-profiling/
https://gperftools.github.io/gperftools/heapprofile.html
二、google-perftools工具下载路径:
https://sourceforge.net/projects/goog-perftools/
gperftools-master.zip <见百度云常用工具>
三、工具使用方法
gperftools适合发现运行过程中内存差异,提供泄露的函数范围。而且进程响影响小,适合高压力快速复现。
详细使用方法请参考http://docs.ceph.org.cn/rados/troubleshooting/memory-profiling/,下面仅对该文档做一些补充说明
1、单独查看一个内存堆分析文件
pprof --text /usr/bin/ceph-osd /var/log/ceph/osd.25.profile.0003.heap
2、比较两个内存堆分析文件
pprof --text --base /var/log/ceph/osd.34.profile.0011.heap /usr/bin/ceph-osd /var/log/ceph/osd.34.profile.0029.heap
3、输出结果说明
[root@node4 gperftools-master]# pprof --text /usr/bin/ceph-osd /var/log/ceph/osd.34.profile.0011.heap |less
Using local file /usr/bin/ceph-osd.
Using local file /var/log/ceph/osd.34.profile.0011.heap.
Total: 3.4 MB
3.3 94.8% 94.8% 3.3 94.8% ceph::logging::Log::create_entry
0.1 1.7% 96.6% 0.1 1.7% AsyncConnection::AsyncConnection
0.0 0.7% 97.2% 0.0 0.7% ceph::buffer::list::append@a34000
0.0 0.6% 97.8% 0.0 0.6% ceph::buffer::create_aligned
0.0 0.5% 98.3% 0.1 1.5% AsyncMessenger::add_accept
0.0 0.4% 98.7% 0.0 0.4% EventCenter::create_time_event
0.0 0.3% 99.0% 0.0 1.0% AsyncMessenger::create_connect
0.0 0.3% 99.2% 2.9 83.2% EventCenter::process_events
0.0 0.1% 99.4% 0.0 0.1% ceph::buffer::list::append@a34490
0.0 0.1% 99.5% 0.5 15.3% OSD::heartbeat
0.0 0.1% 99.6% 0.0 0.1% decode_message
0.0 0.1% 99.7% 0.0 0.1% std::basic_string::_Rep::_S_create
0.0 0.0% 99.7% 0.0 0.0% std::_Rb_tree::_M_emplace_hint_unique
0.0 0.0% 99.7% 0.0 0.1% OSD::ms_verify_authorizer
0.0 0.0% 99.8% 0.0 0.0% std::_Rb_tree::_M_copy
0.0 0.0% 99.8% 0.0 0.0% std::_Rb_tree::M_insert
0.0 0.0% 99.8% 1.0 27.9% AsyncConnection::send_message
0.0 0.0% 99.8% 0.0 0.0% PORT_Alloc_Util
0.0 0.0% 99.9% 0.0 0.0% CryptoAES::get_key_handler
0.0 0.0% 99.9% 0.0 0.0% std::vector::_M_emplace_back_aux
0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::create_static
0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::list::append@a2e290
0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::list::append@a2e3c0
Total:后面显示总共申请的内存
第一列值显示的是该函数内,不包括调用的其他函数,总共申请的堆内存
第四列值显示的是该函数内,包括自己申请的堆内存,和调用外部函数申请的堆内存
第二列和第五列,是第一列和第四例相对于Total的比值
第六列显示具体申请堆内存的函数
4、堆内存释放方法
a、首先使用命令:ceph tell {daemon-type}.{daemon-id} heap stats 查看是否有未使用的堆内存。(注意这里的daemon-type可以是mon、osd、mds,daemon-id是对应服务编号)
举例:
[root@node1 bn_cli]# ceph tell osd.4 heap stats
osd.4 tcmalloc heap stats:------------------------------------------------
MALLOC: 21704640 ( 20.7 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 1406560 ( 1.3 MiB) Bytes in central cache freelist
MALLOC: + 1310656 ( 1.2 MiB) Bytes in transfer cache freelist
MALLOC: + 3316256 ( 3.2 MiB) Bytes in thread cache freelists
MALLOC: + 1175712 ( 1.1 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 28913824 ( 27.6 MiB) Actual memory used (physical + swap)
MALLOC: + 573440 ( 0.5 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 29487264 ( 28.1 MiB) Virtual address space used
MALLOC:
MALLOC: 1039 Spans in use
MALLOC: 36 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
查看是否有:Bytes in page heap freelist (未释放的堆内存)
b、如果Bytes in page heap freelist 不为0,则执行如下命令,可以释放掉未使用的堆内存
ceph tell osd.4 heap release