kudu tserver占用内存过高后会拒绝部分写请求,日志如下:
19/06/01 13:34:12 INFO AsyncKuduClient: Invalidating location 34b1c13d04664cc8bae6689d39b08b77($kudu_tserver:7050) for tablet 858c055c456549569af77d14eaf997e5: Service unavailable: Soft memory limit exceeded (at 92.36% of capacity). See https://kudu.apache.org/releases/1.7.0-cdh5.16.1/docs/troubleshooting.html
1 查看tserver内存详细占用
http://$kudu_tserver:8050/mem-trackers
页面分为两个部分:
Process memory usage 为总体占用
Memory usage by subsystem为详细占比,其中乍看比较繁琐,其实是一个树形结构,通过parent来进行级联,
比如root是全部占用为1.6G,root下面logcache占用了277K,再往下看
block_cache-sharded_lru_cache占用246M,server占用1.4G,server下面又分为很多个tablet,除了tablet之外,还有
log_block_manager占用438M;
如果发现某些tablet占用内存较多,想看下是在哪个table中,可以通过如下命令:
2 查看tserver上的所有tablet
sudo -u kudu kudu remote_replica list $kudu_tserver
比如:
Tablet id: e064f41775084680ab269b3cb3c21c76 State: RUNNING Table name: impala::test_db.test_table1 Partition: RANGE (dt) PARTITION 20190531 <= VALUES < 20190532 Estimated on disk size: 30.71M Schema: Schema [ 0:co1[int32 NOT NULL], 1:co2[string NOT NULL], 2:co3[string NOT NULL], 3:co4[string NOT NULL], 4:co5[string NULLABLE] ] ...
关于kudu的内存估算,和数据量大小以及tablet多少以及活跃replica多少都有关系,详见:
https://kudu.apache.org/docs/scaling_guide.html#memory