Hi Folks,
As the results of 32 disks with xfs_repair -n seems no any error shows up.
We currently tried to deploy CentOS 6.6 for testing. (The previous kernel panic was came from Ubuntu).
The CentOS nodes encountered kernel panic with same daemon but the problem may a bit differ.
- It was broken on
xfs_dir2_sf_get_parent_ino+0xa/0x20in Ubuntu. - Here’s the log in CentOS. It’s broken on
xfs_dir2_sf_getdents+0x2a0/0x3a0
<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
<1>IP: [<ffffffffa0362d60>] xfs_dir2_sf_getdents+0x2a0/0x3a0 [xfs]
<4>PGD 1072327067 PUD 1072328067 PMD 0
<4>Oops: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/pci0000:80/0000:80:03.2/0000:83:00.0/host10/port-10:1/expander-10:1/port-10:1:16/end_device-10:1:16/target10:0:25/10:0:25:0/block/sdz/queue/rotational
<4>CPU 17
<4>Modules linked in: xt_conntrack tun xfs exportfs iptable_filter ipt_REDIRECT iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables ip_vs ipv6 libcrc32c iTCO_wdt iTCO_vendor_support ses enclosure igb i2c_algo_bit sb_edac edac_core i2c_i801 i2c_core sg shpchp lpc_ich mfd_core ixgbe dca ptp pps_core mdio power_meter acpi_ipmi ipmi_si ipmi_msghandler ext4 jbd2 mbcache sd_mod crc_t10dif mpt3sas scsi_transport_sas raid_class xhci_hcd ahci wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 4454, comm: swift-object-se Not tainted 2.6.32-504.23.4.el6.x86_64 #1 Silicon Mechanics Storform R518.v5P/X10DRi-T4+
<4>RIP: 0010:[<ffffffffa0362d60>] [<ffffffffa0362d60>] xfs_dir2_sf_getdents+0x2a0/0x3a0 [xfs]
<4>RSP: 0018:ffff880871f6de18 EFLAGS: 00010202
<4>RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
<4>RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00007faa74006203
<4>RBP: ffff880871f6de68 R08: 000000032eb04bc9 R09: 0000000000000004
<4>R10: 0000000000008030 R11: 0000000000000246 R12: 0000000000000000
<4>R13: 0000000000000002 R14: ffff88106eff7000 R15: ffff8808715b4580
<4>FS: 00007faa85425700(0000) GS:ffff880028360000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>CR2: 0000000000000001 CR3: 0000001072325000 CR4: 00000000001407e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process swift-object-se (pid: 4454, threadinfo ffff880871f6c000, task ffff880860f18ab0)
<4>Stack:
<4> ffff880871f6de28 ffffffff811a4bb0 ffff880871f6df38 ffff880874749cc0
<4><d> 0000000100000103 ffff8802381f8c00 ffff880871f6df38 ffff8808715b4580
<4><d> 0000000000000082 ffff8802381f8d88 ffff880871f6dec8 ffffffffa035ab31
<4>Call Trace:
<4> [<ffffffff811a4bb0>] ? filldir+0x0/0xe0
<4> [<ffffffffa035ab31>] xfs_readdir+0xe1/0x130 [xfs]
<4> [<ffffffff811a4bb0>] ? filldir+0x0/0xe0
<4> [<ffffffffa038fe29>] xfs_file_readdir+0x39/0x50 [xfs]
<4> [<ffffffff811a4e30>] vfs_readdir+0xc0/0xe0
<4> [<ffffffff8119bd86>] ? final_putname+0x26/0x50
<4> [<ffffffff811a4fb9>] sys_getdents+0x89/0xf0
<4> [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
<4>Code: 01 00 00 00 48 c7 c6 38 6b 3a a0 48 8b 7d c0 ff 55 b8 85 c0 0f 85 af 00 00 00 49 8b 37 e9 ec fd ff ff 66 0f 1f 84 00 00 00 00 00 <41> 80 7c 24 01 00 0f 84 9c 00 00 00 45 0f b6 44 24 03 41 0f b6
<1>RIP [<ffffffffa0362d60>] xfs_dir2_sf_getdents+0x2a0/0x3a0 [xfs]
<4> RSP <ffff880871f6de18>
<4>CR2: 0000000000000001
PID: 4454 TASK: ffff880860f18ab0 CPU: 17 COMMAND: "swift-object-se"
ROOT: / CWD: /
FD FILE DENTRY INODE TYPE PATH
0 ffff881073604900 ffff8810749a9440 ffff8808740b9728 CHR /dev/null
1 ffff881073604900 ffff8810749a9440 ffff8808740b9728 CHR /dev/null
2 ffff881073604900 ffff8810749a9440 ffff8808740b9728 CHR /dev/null
3 ffff881074222840 ffff88106e184980 ffff88106e16cd48 SOCK
4 ffff881072952c00 ffff88106e1848c0 ffff8810711cca08 SOCK
5 ffff88087154b2c0 ffff88044f0f1780 ffff880018e51a08 SOCK
6 ffff8810716f2600 ffff88107122d5c0 ffff881071315cb8 REG /tmp/ffi1ECJ8Z
7 ffff88086bd0a6c0 ffff88086fc4b840 ffff88086fc4a100 REG /tmp/ffiIArHUO
8 ffff88106f516ec0 ffff881067d9ae00 ffff8808745aa5e8 REG [eventpoll]
9 ffff88106ed35b40 ffff88106e15b200 ffff88106e201cc8 SOCK
10 ffff88106f31ae00 ffff881074ad75c0 ffff88106e169a08 SOCK
11 ffff88106ede2740 ffff881067f9a8c0 ffff8808745aa5e8 REG [eventpoll]
12 ffff880122e8bc80 ffff8808745a8240 ffff881074379d48 CHR /dev/urandom
13 ffff88087162e200 ffff88086fd87200 ffff88086fe82748 SOCK
14 ffff88087135e840 ffff88086fc653c0 ffff88086fe82488 SOCK
15 ffff88106f36f900 ffff88106e263680 ffff8808745aa5e8 REG [eventpoll]
16 ffff8810737f8680 ffff88106e3e9c80 ffff8808745aa5e8 REG [eventpoll]
17 ffff881073635540 ffff88106e1f9240 ffff8808745aa5e8 REG [eventpoll]
18 ffff88106ef5ba40 ffff88105bb6b080 ffff8808745aa5e8 REG [eventpoll]
19 ffff881074222300 ffff88105b975300 ffff8808745aa5e8 REG [eventpoll]
20 ffff881073770f00 ffff881013478080 ffff8808745aa5e8 REG [eventpoll]
21 ffff8810737f8bc0 ffff88106e3e9500 ffff8808745aa5e8 REG [eventpoll]
22 ffff88106ef5bc80 ffff88105bb72e00 ffff8808745aa5e8 REG [eventpoll]
23 ffff88106ef25e00 ffff88106e3e9bc0 ffff8808745aa5e8 REG [eventpoll]
24 ffff881071950d80 ffff8810383ee980 ffff8808745aa5e8 REG [eventpoll]
25 ffff88106ecca600 ffff881067d96840 ffff8808745aa5e8 REG [eventpoll]
26 ffff8808737ec740 ffff880855d95cc0 ffff8808745aa5e8 REG [eventpoll]
27 ffff88107345d3c0 ffff880fc46160c0 ffff8808745aa5e8 REG [eventpoll]
28 ffff88086bf2d600 ffff880777987b00 ffff880159ac0448 SOCK
29 ffff8808737e9240 ffff880855c80a40 ffff8808745aa5e8 REG [eventpoll]
30 ffff88106f5e0140 ffff880ff5752440 ffff8808745aa5e8 REG [eventpoll]
31 ffff8808703a19c0 ffff8807847c8e40 ffff8808745aa5e8 REG [eventpoll]
32 ffff88086bd738c0 ffff88033be10800 ffff8806b47f2c08 SOCK
33 ffff88087119eb40 ffff8806916b48c0 ffff8804cd27e648 SOCK
34 ffff880870aed480 ffff8806b3fc4900 ffff880015583588 REG /srv/node/d199/objects/12860/2c0/323cc020fd7dbd6c12472cd1c10742c0/1436266036.98015.ts
35 ffff88106eeb0e00 ffff88101347de40 ffff8808745aa5e8 REG [eventpoll]
36 ffff8808703ed6c0 ffff88086fd65540 ffff8805eb03ed88 REG /srv/node/d205/quarantined/objects/cd1d68f515006d443a54ff4f658091bc-a114bba1449b45238abf38dc741d7c27/1436254020.89801.ts
37 ffff8810718343c0 ffff88105b9d32c0 ffff8808745aa5e8 REG [eventpoll]
38 ffff8808713da780 ffff880010c9a900 ffff88096368a188 REG /srv/node/d224/quarantined/objects/b146865bf8034bfc42570b747c341b32/1436266042.57775.ts
39 ffff880871cb03c0 ffff880495a8b380 ffff8808a5e6c988 REG /srv/node/d224/tmp/tmpSpnrHg
40 ffff8808715b4540 ffff8804819c58c0 ffff8802381f8d88 DIR /srv/node/d224/quarantined/objects/b146865bf8034bfc42570b747c341b32
41 ffff880871fce240 ffff880951136c00 ffff880bacf63d88 DIR /srv/node/d199/objects/12860/2c0/323cc020fd7dbd6c12472cd1c10742c0
I’ve got the vmcore dump from operator. Does vmcore help for troubleshooting kind issue ?
Thanks // Hugo
2015-06-18 22:59 GMT+08:00 Eric Sandeen <sandeen@xxxxxxxxxxx>:
On 6/18/15 9:29 AM, Kuo Hugo wrote:
>>- Have you tried an 'xfs_repair -n' of the affected filesystem? Note that -n will report problems only and prevent any modification by repair.
>
> *We might to to xfs_repair if we can address which disk causes the issue. *
If you do, please save the output, and if it finds anything, please provide the output in this thread.
Thanks,
-Eric
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
