Linux SI 问题处理总结-重要

Linux SI 问题处理总结
系统死机问题（Crash issue）
1.内核/驱动Oops
方法一：用addr2line 分析 Oops Callstack，

方法二：在__do_kernel_fault()，使用CVD设置断点。
方法三：在Kernal_restart(), 打印all thread callstack。
方法四：用objdump 反编译分析

[ 922.768338] Unable to handle kernel paging request at virtual address 6b6b6dc7
[ 922.775453] pgd = d2bec000
[ 922.778138] [6b6b6dc7] *pgd=00000000
[ 922.781698] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 922.786990] Modules linked in: mtk_mmap(PO) usbhid ntfs fuse dma_buf_test_exporter mali mt7603u_sta(O) mtk_mod(PO) fusion(O)
[ 922.798190] CPU: 2 PID: 2983 Comm: wpa_supplicant Tainted: P O 3.18.25 #15
[ 922.806086] task: c0dce400 ti: c0946000 task.ti: c0946000
[ 922.811473] PC is at nl80211_post_doit+0x38/0x88
[ 922.816069] LR is at genl_rcv_msg+0x270/0x2e8
[ 922.820408] pc : [<c03b1ab8>] lr : [<c031afac>] psr: 200f0093
[ 922.820408] sp : c0947cd0 ip : 00000000 fp : 00000000
[ 922.831862] r10: 00000014 r9 : 00000000 r8 : d2013900
[ 922.837070] r7 : d253b980 r6 : c06eba54 r5 : d253b980 r4 : c043b03c
[ 922.843579] r3 : 6b6b6b6b r2 : 00000014 r1 : 200f0013 r0 : c043b03c
[ 922.850090] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[ 922.857293] Control: 10c5383d Table: 12bec06a DAC: 00000015
[ 922.863019]
[ 922.863019] PC: 0xc03b1a38:
[ 922.867271] 1a38 e59332d0 e3530000 0afffffa e1d330b4 e3010388 e1530000 8a000005 e2840e16
[ 922.875429] 1a58 e12fff3c e3500000 05953004 05843120 e8bd8038 e3e00021 e8bd8038 e3e0000f
[ 922.883587] 1a78 e8bd8038 c052c5be e92d4038 e1a04000 e592301c e1a05001 e3530000 0a00000d
[ 922.891746] 1a98 e5d02011 e3120010 0a000002 e5933010 e3530000 0a000007 e10f1000 f10c0080
[ 922.899904] 1ab8 e593325c ee1d2f90 e7920003 e2400001 e7820003 e121f001 e5d43011 e3130004
[ 922.908063] 1ad8 0a000000 ebfd6843 e5d43011 e3130020 08bd8038 e59500a0 e5901000 e2511010
[ 922.916221] 1af8 08bd8038 e2800010 e8bd4038 eaf81e8d e92d407f e1a04001 e5915018 e591601c
[ 922.924379] 1b18 e5953000 e5933128 e3530000 0a00001b e5913014 e59322c4 e3520000 0a000019
[ 922.932539]
[ 922.932539] LR: 0xc031af2c:
[ 922.936791] af2c e2509000 ba000024 e5963008 e28d0024 e58d6014 e3a01008 e58d8020 e58d300c
[ 922.944949] af4c e5973024 e58d3010 e2863010 e58d3018 e2866014 e5973010 e58d601c e58d302c
[ 922.953108] af6c ebfa7973 e5943024 e3530000 1a00000c e5953004 e1a00007 e28d100c e12fff33
[ 922.961266] af8c e5943028 e1a09000 e3530000 0a00000a e1a00005 e1a01007 e28d200c e12fff33
[ 922.969424] afac ea000005 e1a00005 e1a01007 e28d200c e12fff33 e2509000 0affffec e5d43021
[ 922.977583] afcc e3530000 0a000006 e1a00008 ebf67f57 ea000003 e3e05015 ea000002 e3e0500b
[ 922.985741] afec ea000000 e1a05009 e5d43021 e3530000 1a000002 ebfffdcd ea000000 e3e05001
[ 922.993899] b00c e1a00005 e28dd030 e8bd87f0 c05e1200 c031a780 c031a748 e92d4037 e1a04001
[ 923.002059]
[ 923.002059] SP: 0xc0947c50:
[ 923.006311] 7c50 c0de4600 d2013900 00000000 00000014 00000000 bfbec56c d7995000 c03b1ab8
[ 923.014470] 7c70 200f0093 ffffffff c0947cbc d2013900 00000000 c0408b98 c043b03c 200f0013
[ 923.022627] 7c90 00000014 6b6b6b6b c043b03c d253b980 c06eba54 d253b980 d2013900 00000000
[ 923.030786] 7cb0 00000014 00000000 00000000 c0947cd0 c031afac c03b1ab8 200f0093 ffffffff
[ 923.038944] 7cd0 c03b1a80 c05e6b38 c043b03c c031afac c043ac3e c003e7e8 00000001 5b432d90
[ 923.047103] 7cf0 c1000ba7 c06eba40 c06eba50 c06eba54 d2013900 ce534280 cebd2600 d2013440
[ 923.055261] 7d10 c06eba40 d253b980 c031ad3c 0000001c c0947d54 00000000 d253b980 c031a520
[ 923.063420] 7d30 c05e290c d253b980 c09b84c0 c031ad28 c031ad08 d2013440 d253b980 c0319ed0
[ 923.071579]
[ 923.071579] R0: 0xc043afbc:
[ 923.075832] afbc 00030000 00030000 00080000 00000000 000b0000 00030020 00000000 c043ac3e
[ 923.083989] afdc c03bbb88 c03bbc34 c03b282c 00000501 c043ac3e c03bea90 00000000 00000000
[ 923.092147] affc 00010402 c043ac3e c03b5610 c03b5530 00000000 00001405 c043ac3e c03bbee4
[ 923.100306] b01c 00000000 00000000 00010606 c043ac3e c03bffc0 00000000 00000000 00010507
[ 923.108465] b03c c043ac3e c03b10fc 00000000 00000000 00011408 c043ac3e c03b3418 00000000
[ 923.116623] b05c 00000000 00010e09 c043ac3e c03bd9b8 00000000 00000000 00012e0a c043ac3e
[ 923.124782] b07c c03bdaf0 00000000 00000000 00012e0b c043ac3e c03bd87c 00000000 00000000
[ 923.132940] b09c 00010e0c c043ac3e c03bd2bc 00000000 00000000 00010e0e c043ac3e c03c0568
[ 923.141099]
[ 923.141099] R4: 0xc043afbc:
[ 923.145351] afbc 00030000 00030000 00080000 00000000 000b0000 00030020 00000000 c043ac3e
[ 923.153509] afdc c03bbb88 c03bbc34 c03b282c 00000501 c043ac3e c03bea90 00000000 00000000
[ 923.161668] affc 00010402 c043ac3e c03b5610 c03b5530 00000000 00001405 c043ac3e c03bbee4
[ 923.169826] b01c 00000000 00000000 00010606 c043ac3e c03bffc0 00000000 00000000 00010507
[ 923.177984] b03c c043ac3e c03b10fc 00000000 00000000 00011408 c043ac3e c03b3418 00000000
[ 923.186143] b05c 00000000 00010e09 c043ac3e c03bd9b8 00000000 00000000 00012e0a c043ac3e
[ 923.194302] b07c c03bdaf0 00000000 00000000 00012e0b c043ac3e c03bd87c 00000000 00000000
[ 923.202460] b09c 00010e0c c043ac3e c03bd2bc 00000000 00000000 00010e0e c043ac3e c03c0568
[ 923.210619]
[ 923.210619] R5: 0xd253b900:
[ 923.214871] b900 00000ba7 0001e62f c0319ed8 c0319ed8 c031a364 c02e9d8c c02eadf4 c02ebdb4
[ 923.223029] b920 c000e120 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.231188] b940 00000000 00000000 00000000 00000001 00000ba7 0001e629 5a5a5a5a 5a5a5a5a
[ 923.239347] b960 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a
[ 923.247505] b980 00000000 00000000 00000000 00000000 d2013440 00000000 00000ba7 00000000
[ 923.255663] b9a0 00000000 c1000ba7 00000000 00000008 c09b84c0 00000000 00000000 00000000
[ 923.263822] b9c0 00000000 00000000 00000000 c0317950 00000000 0000001c 00000000 00000000
[ 923.271980] b9e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.280140]
[ 923.280140] R6: 0xc06eb9d4:
[ 923.284391] b9d4 0001ba42 bf00a9ec bf004420 c00cced4 c00ccfc8 c000e120 00000000 00000000
[ 923.292550] b9f4 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.300708] ba14 00000000 00000000 00000003 000002f1 0001a5e9 5a5a5a5a 5a5a5a5a 5a5a5a5a
[ 923.308867] ba34 5a5a5a5a 5a5a5a5a 5a5a5a5a 0000001c 00050013 5b432d90 c1000ba7 00000008
[ 923.317025] ba54 00030008 00000006 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b
[ 923.325183] ba74 6b6b6b6b 6b6b6b6b 6b6b6b6b 00000000 00000000 00000000 00000000 00000000
[ 923.333342] ba94 00000000 00000000 00000000 00000001 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b
[ 923.341500] bab4 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b 6b6b6b6b
[ 923.349660]
[ 923.349660] R7: 0xd253b900:
[ 923.353912] b900 00000ba7 0001e62f c0319ed8 c0319ed8 c031a364 c02e9d8c c02eadf4 c02ebdb4
[ 923.362070] b920 c000e120 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.370228] b940 00000000 00000000 00000000 00000001 00000ba7 0001e629 5a5a5a5a 5a5a5a5a
[ 923.378387] b960 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a
[ 923.386545] b980 00000000 00000000 00000000 00000000 d2013440 00000000 00000ba7 00000000
[ 923.394703] b9a0 00000000 c1000ba7 00000000 00000008 c09b84c0 00000000 00000000 00000000
[ 923.402862] b9c0 00000000 00000000 00000000 c0317950 00000000 0000001c 00000000 00000000
[ 923.411020] b9e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.419180]
[ 923.419180] R8: 0xd2013880:
[ 923.423432] 3880 00000000 00000000 00000000 00000002 00000001 ffff1630 c02f0544 c03187b8
[ 923.431590] 38a0 c0318880 c031a020 c030d2d4 c0301bec c0301c54 c0593c0c c02f8bc8 c02f8c60
[ 923.439748] 38c0 c02f8cfc c0598908 c000897c c0580cfc c03fdb58 c000e1c0 00000000 00000002
[ 923.447907] 38e0 00000001 ffff1630 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a 5a5a5a5a
[ 923.456065] 3900 00000000 00000000 00000000 c06eba54 00000000 00000000 00000000 00000000
[ 923.464224] 3920 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.472382] 3940 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.480541] 3960 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.488702] Process wpa_supplicant (pid: 2983, stack limit = 0xc0946238)
[ 923.495385] Stack: (0xc0947cd0 to 0xc0948000)
[ 923.499727] 7cc0: c03b1a80 c05e6b38 c043b03c c031afac
[ 923.507887] 7ce0: c043ac3e c003e7e8 00000001 5b432d90 c1000ba7 c06eba40 c06eba50 c06eba54
[ 923.516045] 7d00: d2013900 ce534280 cebd2600 d2013440 c06eba40 d253b980 c031ad3c 0000001c
[ 923.524203] 7d20: c0947d54 00000000 d253b980 c031a520 c05e290c d253b980 c09b84c0 c031ad28
[ 923.532363] 7d40: c031ad08 d2013440 d253b980 c0319ed0 c09b84c0 7fffffff 00000000 00000000
[ 923.540520] 7d60: c09b84c0 c0947f7c 00000000 0000001c c0947e10 c031a364 c0947fb0 00000020
[ 923.548678] 7d80: c057bde0 00000000 00000008 00000000 c0947fb0 c5437700 00000000 00000ba7
[ 923.556836] 7da0: 00000000 00000000 0013a428 d141af40 0000001c c0947f7c c0dce400 00000000
[ 923.564995] 7dc0: d141af40 c0947e88 00000000 c02e9d8c 00000000 00000000 00000000 c0947e10
[ 923.573153] 7de0: c0dce400 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 923.581312] 7e00: 00000000 00000000 00000000 00000000 d1c63998 c0011264 c0947e48 0000001c
[ 923.589481] 7e20: d141af40 00000000 c0947d94 c0947f7c c0947ec8 c0947e88 c0947ec8 00000128
[ 923.597628] 7e40: 00000000 c0947e88 00000008 00000000 c0947f7c c0947f7c 0000001c 00000000
[ 923.605788] 7e60: c0947e68 c02eadf4 d2028780 d2876c40 d1c63998 d2028780 80070d00 d1c63998
[ 923.613946] 7e80: c05b1540 d2876c40 0013d73c 00000000 00000001 c04014fc c0947eb4 200f0013
[ 923.622105] 7ea0: 000080d0 00000000 d1c63998 c0946018 000000d0 200f0013 c0946000 d1904640
[ 923.630263] 7ec0: c0946000 001a0017 00000010 00000000 00000000 c02ee63c 00000001 c00bad00
[ 923.638421] 7ee0: d203c310 001a0017 c0946000 d2493cc0 c05b1e00 d2887200 12069000 d2028780
[ 923.646580] 7f00: c0946000 d1c63998 c05b1540 d2876c40 12069000 c00d077c 00000001 c00bad00
[ 923.654738] 7f20: d2028780 d1c63998 d1c639c4 00000000 00000000 d2493ce0 d1c639e8 d203c310
[ 923.662897] 7f40: c613fc88 00000000 bea97714 d141af40 00000000 bea97714 00000128 c000e2a4
[ 923.671055] 7f60: c0946000 00000000 0000000c c02ebdb4 c0dce400 00000000 fffffff7 c0947ec8
[ 923.679214] 7f80: 0000000c c0947e88 00000001 00000000 00000000 00000000 00115ff0 00115f78
[ 923.687371] 7fa0: b6f8b990 c000e120 00115ff0 00115f78 00000029 bea97714 00000000 00000000
[ 923.695530] 7fc0: 00115ff0 00115f78 b6f8b990 00000128 bea97714 00000001 ffffeec4 0000000c
[ 923.703689] 7fe0: 00000000 bea976cc b6f78dd4 b6ce0afc 600f0010 00000029 dead0118 dead0118
[ 923.711863] [<c03b1ab8>] (nl80211_post_doit) from [<c031afac>] (genl_rcv_msg+0x270/0x2e8)
[ 923.720009] [<c031afac>] (genl_rcv_msg) from [<c031a520>] (netlink_rcv_skb+0x58/0xac)
[ 923.727818] [<c031a520>] (netlink_rcv_skb) from [<c031ad28>] (genl_rcv+0x20/0x34)
[ 923.735282] [<c031ad28>] (genl_rcv) from [<c0319ed0>] (netlink_unicast+0x128/0x214)
[ 923.742921] [<c0319ed0>] (netlink_unicast) from [<c031a364>] (netlink_sendmsg+0x30c/0x374)
[ 923.751166] [<c031a364>] (netlink_sendmsg) from [<c02e9d8c>] (sock_sendmsg+0x70/0x8c)
[ 923.758977] [<c02e9d8c>] (sock_sendmsg) from [<c02eadf4>] (___sys_sendmsg.part.3+0x17c/0x200)
[ 923.767484] [<c02eadf4>] (___sys_sendmsg.part.3) from [<c02ebdb4>] (__sys_sendmsg+0x4c/0x70)
[ 923.775905] [<c02ebdb4>] (__sys_sendmsg) from [<c000e120>] (ret_fast_syscall+0x0/0x38)
[ 923.783798] Code: e3530000 0a000007 e10f1000 f10c0080 (e593325c)
[ 923.789866] Disable IRQ, CPU loops forever.
[ 923.794032] let go panic
[ 923.796554] ---[ end trace 39988029c654e312 ]---

2.Apps/user层死机问题。
方法一：应用层的SIGSEGV,用CVD在__do_user_fault()设置断点。
方法二：用GDB 分析coredump，
   (1):echo /3rd_rw/core/core_%e_sig%s_pid%p>/proc/sys/kernel/core_pattern
   (2):ulimit -c unlimited
   (3):查询coredump文件是否存在
   (4):./GDB bin
   (5):bt 或者 wherecoredump
   (6):使用addr2line定位file和line



   参考：http://mtkdtvwiki01/wiki/index.php/Sony_2K15_Sakura_project_debug#GDB_on_Android.2C_system_hang_issue_check
方法三用libarm_sig_tracer.so 打印callstack
   (1): libarm_sig_tracer.so放在/basic/分区
   (2):复制问题
   (3):分析打印出来callstack
   (4):使用addr2line定位file和line

080 2018] [dtv_app_mtk]>(SIGNAL TRACER)PROCESS SIGNAL
[Wed Aug 15 13:27:35.080 2018] [dtv_app_mtk]>(SIGNAL TRACER)Signal 11 catched, dumping...
[Wed Aug 15 13:27:35.090 2018] [dtv_app_mtk]>->0xb2045ea8(/basic/libmtkapp.so+0x006dbea8)
[Wed Aug 15 13:27:35.090 2018] [dtv_app_mtk]>   ->R00:0x00000000 R01:0x00000021 R02:0x00000000 R03:0x8F25C034
[Wed Aug 15 13:27:35.100 2018] [dtv_app_mtk]>   ->R04:0xB299A994 R05:0x00000000 R06:0x84D5D9F4 R07:0x00000152
[Wed Aug 15 13:27:35.110 2018] [dtv_app_mtk]>   ->R08:0x8155DF90 R09:0x84D5D9F4 R10:0x8155E684 R11:0x8155DB24
[Wed Aug 15 13:27:35.110 2018] [dtv_app_mtk]>   ->R12:0x8155DA20 R13:0x8155DAE4 R14:0xB2043DF8 R15:0xB2043DF8
[Wed Aug 15 13:27:35.120 2018] [dtv_app_mtk]>
[Wed Aug 15 13:27:35.120 2018] [dtv_app_mtk]>->0xb2043df8(/basic/libmtkapp.so+0x006d9df8)
[Wed Aug 15 13:27:35.130 2018] [dtv_app_mtk]>   ->R00:0x00000000 R01:0x00000021 R02:0x00000000 R03:0x8F25C034
[Wed Aug 15 13:27:35.130 2018] [dtv_app_mtk]>   ->R04:0xB299A994 R05:0x00000000 R06:0x84D5D9F4 R07:0x00000152
[Wed Aug 15 13:27:35.140 2018] [dtv_app_mtk]>   ->R08:0x8155DF90 R09:0x84D5D9F4 R10:0x8155E684 R11:0x8155DBE4
[Wed Aug 15 13:27:35.150 2018] [dtv_app_mtk]>   ->R12:0x8155DA20 R13:0x8155DB24 R14:0xB2043AAC R15:0xB2043AAC
[Wed Aug 15 13:27:35.150 2018] [dtv_app_mtk]>
[Wed Aug 15 13:27:35.160 2018] [dtv_app_mtk]>->0xb2043aac(/basic/libmtkapp.so+0x006d9aac)
[Wed Aug 15 13:27:35.160 2018] [dtv_app_mtk]>   ->R00:0x00000000 R01:0x00000021 R02:0x00000000 R03:0x8F25C034
[Wed Aug 15 13:27:35.170 2018] [dtv_app_mtk]>   ->R04:0xB299A994 R05:0x00000000 R06:0x84D5D9F4 R07:0x00000152
[Wed Aug 15 13:27:35.170 2018] [dtv_app_mtk]>   ->R08:0x8155DF90 R09:0x84D5D9F4 R10:0x8155E684 R11:0x8155DC44
[Wed Aug 15 13:27:35.180 2018] [dtv_app_mtk]>   ->R12:0x8155DA20 R13:0x8155DBE4 R14:0xB1CC5B24 R15:0xB1CC5B24
[Wed Aug 15 13:27:35.190 2018] [dtv_app_mtk]>
[Wed Aug 15 13:27:35.190 2018] [dtv_app_mtk]>->0xb1cc5b24(nav_brdcst_msg_to_components + 0x00000264)(/basic/libmtkapp.so+0x0035bb24)
[Wed Aug 15 13:27:35.200 2018] [dtv_app_mtk]>   ->R00:0x00000000 R01:0x00000021 R02:0x00000000 R03:0x8F25C034
[Wed Aug 15 13:27:35.210 2018] [dtv_app_mtk]>   ->R04:0xB299A994 R05:0x00000000 R06:0x84D5D9F4 R07:0x00000152
[Wed Aug 15 13:27:35.210 2018] [dtv_app_mtk]>   ->R08:0x8155DF90 R09:0x84D5D9F4 R10:0x8155E684 R11:0x8155DC6C
[Wed Aug 15 13:27:35.220 2018] [dtv_app_mtk]>   ->R12:0x8155DA20 R13:0x8155DC44 R14:0xB1CC13DC R15:0xB1CC13DC

内存泄露（Memory leak）
1.系统内核 memory leack
   (1):Enabel kernel config
   CONFIG_DEBUG_KMEMLEAK=y
   CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=2000
   (2):echo clear > /sys/kernel/debug/kmemleak
   (3):start stressing (with cat /proc/meminfo) for several hours)(#!/system/bin/shwhile((1)):docat /proc/meminfo done& )
   (4):echo a > /sys/power/wake_lock
   (5):echo scan > /sys/kernel/debug/kmemleak
   (6):cat /sys/kernel/debug/kmemleak > /3rd_rw/kmemleak.log
   (7):分析kmemleak.log

2.应用层 memroyleak
（1）top -a/top check RES项是否一直变大。

（2）在线程里添加mtrace 实现代码：

（3）输入两次kill -30 pid ，会在/tmp/mnt/路径下生成一个mtrace文件。
（4）usr/bin/mtrace dtv_svc dtv_svc_trace.log >dump.log
（5）打开dumo.log如下所示,就能看出哪些文件哪行code存在泄露，-》x_raw_sect_filter.c文件 552行存在泄露

3.Dtv_svc handle 泄露 (参考http://wiki.mediatek.inc/pages/viewpage.action?pageId=376931466)
mw.handle.s ==> Returns handle statistics
mw.handle.dg <group_id> ==> Dump all handle for <group_id>
mw.handle.sg <group_id> ==> Print callstack for handle allocated in <group_id>
Log中有对应的fail/error log表明create handle失败: {SM-SCC} [SCC] _scc.c:856 rm_open_comp() fail return -6 ==> -6表明allocate handle的时候失败了
Step 1: 复制前后分别输入如下command, 对比两次的结果，找出handle数目变化较大的group(ex, id1);
mw.handle.s
Step 2: 重新复制，复制前输入如下command
mw.handle.dg id1
mw.handle.sg id1
Step 3: 复制一段时间后，输入如下command, 对比两次dg出来的结果，找出新增的handle.
mw.handle.dg id1
Step 4: 在log中search这些新增的handle的callstack，若很多都是相同的callstack所触发，则很有可能就是这里有leakage问题
Step 5: 结合source code做对应的分析

关键是user 层backtrace 的打印

Linux SI 问题处理总结-重要

猜你喜欢