I am testing openshift with ceph rbd, it works as expected. except that sometimes the container which has a rbd volume start slowly. And the load on the node that containers running will pretty high, until following error raise in dmesg.
After some google, i found one similar issue at[0]. seems it is a kernel bug? But since i can not reproduce this issue steadily, so i wanna make sure that does anyone could confirm this issue and fix it?
Btw, here is my env:
OS: centos 7.5
kernel: Linux ocm-74 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
ceph: ceph-12.2.5-0.el7.x86_64 from ceph offical repo
openshift: origin-3.9.0-1.el7.git.0.ba7faec.x86_64
4381870.921579] device veth816c5e2f entered promiscuous mode
[4381899.771170] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [mount:2760216]
[4381899.772326] Modules linked in: vfat fat isofs ip_vs fuse ext4 mbcache jbd2 rbd libceph dns_resolver cfg80211 rfkill udp_diag unix_diag tcp_diag inet_diag veth nf_conntrack_netlink nfnetlink xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_mark xt_comment ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay(T) scsi_transport_iscsi bonding vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_ssif joydev mei_me mei iTCO_wdt iTCO_vendor_support
[4381899.772379] pcspkr dcdbas ipmi_si ipmi_devintf ipmi_msghandler shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod sd_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ixgbe libata crc32c_intel tg3 megaraid_sas i2c_core mdio dca ptp pps_core dm_mirror dm_region_hash dm_log dm_snapshot target_core_user uio target_core_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common dm_multipath dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_mod libcrc32c
[4381899.772421] CPU: 14 PID: 2760216 Comm: mount Kdump: loaded Tainted: G W L ------------ T 3.10.0-862.6.3.el7.x86_64 #1
[4381899.772423] Hardware name: Dell Inc. PowerEdge R720/0DCWD1, BIOS 2.6.1 02/12/2018
[4381899.772426] task: ffff93917178dee0 ti: ffff93a1cf22c000 task.ti: ffff93a1cf22c000
[4381899.772428] RIP: 0010:[<ffffffffac14a158>] [<ffffffffac14a158>] __call_rcu+0x98/0x2c0
[4381899.772440] RSP: 0018:ffff93a1cf22fd30 EFLAGS: 00000246
[4381899.772441] RAX: 0000000002e07679 RBX: ffff939c3f9dbb80 RCX: ffffffffacd41e20
[4381899.772443] RDX: ffffffffacc73000 RSI: 0000000000014340 RDI: 0000000000000246
[4381899.772445] RBP: ffff93a1cf22fd58 R08: 0000000000000000 R09: 0000000000000000
[4381899.772446] R10: ffff939c3f9dbb80 R11: ffffdb1b021f3800 R12: 000000002a7c93a8
[4381899.772448] R13: ffff93a1cf22fd58 R14: ffff93a1cf22fcb0 R15: ffff938cc7ce208f
[4381899.772450] FS: 00007fc6c57db880(0000) GS:ffff939c3f9c0000(0000) knlGS:0000000000000000
[4381899.772452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4381899.772465] CR2: 00007fc6c499c15c CR3: 0000001084400000 CR4: 00000000001607e0
[4381899.772467] Call Trace:
[4381899.772474] [<ffffffffac14a39d>] call_rcu_sched+0x1d/0x20
[4381899.772479] [<ffffffffac2333cf>] d_free+0x4f/0x70
[4381899.772481] [<ffffffffac2339ea>] __dentry_kill+0x16a/0x180
[4381899.772483] [<ffffffffac233cbe>] shrink_dentry_list+0xde/0x230
[4381899.772485] [<ffffffffac233eaa>] shrink_dcache_sb+0x9a/0xe0
[4381899.772491] [<ffffffffac21fa71>] do_remount_sb+0x51/0x200
[4381899.772496] [<ffffffffac240047>] do_mount+0x757/0xce0
[4381899.772501] [<ffffffffac1b3b82>] ? memdup_user+0x42/0x70
[4381899.772503] [<ffffffffac240913>] SyS_mount+0x83/0xd0
[4381899.772512] [<ffffffffac720795>] system_call_fastpath+0x1c/0x21
[4381899.772513] Code: 3c cd a0 53 d3 ac 80 3d 66 eb 02 01 00 8b 87 70 01 00 00 0f 85 3a 01 00 00 80 3d 3f ba bc 00 00 0f 84 bd 01 00 00 4c 89 ef 57 9d <0f> 1f 44 00 00 48 83 c4 10 5b 41 5c 41 5d 5d c3 0f 1f 84 00 00
[4381899.984579] rbd: rbd0: encountered watch error: -107
[4381927.770763] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [mount:2760216]
[4381927.771950] Modules linked in: vfat fat isofs ip_vs fuse ext4 mbcache jbd2 rbd libceph dns_resolver cfg80211 rfkill udp_diag unix_diag tcp_diag inet_diag veth nf_conntrack_netlink nfnetlink xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_mark xt_comment ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay(T) scsi_transport_iscsi bonding vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_ssif joydev mei_me mei iTCO_wdt iTCO_vendor_support
[4381927.772003] pcspkr dcdbas ipmi_si ipmi_devintf ipmi_msghandler shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod sd_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ixgbe libata crc32c_intel tg3 megaraid_sas i2c_core mdio dca ptp pps_core dm_mirror dm_region_hash dm_log dm_snapshot target_core_user uio target_core_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common dm_multipath dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_mod libcrc32c
[4381927.772045] CPU: 14 PID: 2760216 Comm: mount Kdump: loaded Tainted: G W L ------------ T 3.10.0-862.6.3.el7.x86_64 #1
[4381927.772058] Hardware name: Dell Inc. PowerEdge R720/0DCWD1, BIOS 2.6.1 02/12/2018
[4381927.772062] task: ffff93917178dee0 ti: ffff93a1cf22c000 task.ti: ffff93a1cf22c000
[4381927.772063] RIP: 0010:[<ffffffffac7167f0>] [<ffffffffac7167f0>] _raw_spin_lock+0x10/0x30
[4381927.772073] RSP: 0018:ffff93a1cf22fdb8 EFLAGS: 00000246
[4381927.772075] RAX: 0000000000000000 RBX: ffff939c3f9dbb80 RCX: 00fb22a600000000
[4381927.772076] RDX: 0000000000000001 RSI: 0000000000fb22a6 RDI: ffff93905e0a2298
[4381927.772078] RBP: ffff93a1cf22fdf8 R08: 0000000000000000 R09: 0000000000000000
[4381927.772079] R10: ffff939c3f9dbb80 R11: ffffdb1b021f3800 R12: ffff93a1cf22fd30
[4381927.772081] R13: 0000000000000246 R14: 0000000000000010 R15: ffffffffac14a158
[4381927.772083] FS: 00007fc6c57db880(0000) GS:ffff939c3f9c0000(0000) knlGS:0000000000000000
[4381927.772085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4381927.772086] CR2: 00007fc6c499c15c CR3: 0000001084400000 CR4: 00000000001607e0
[4381927.772088] Call Trace:
[4381927.772097] [<ffffffffac233c1c>] ? shrink_dentry_list+0x3c/0x230
[4381927.772100] [<ffffffffac233eaa>] shrink_dcache_sb+0x9a/0xe0
[4381927.772106] [<ffffffffac21fa71>] do_remount_sb+0x51/0x200
[4381927.772111] [<ffffffffac240047>] do_mount+0x757/0xce0
[4381927.772116] [<ffffffffac1b3b82>] ? memdup_user+0x42/0x70
[4381927.772118] [<ffffffffac240913>] SyS_mount+0x83/0xd0
[4381927.772124] [<ffffffffac720795>] system_call_fastpath+0x1c/0x21
[4381927.772125] Code: 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 c5 2c ff ff 5d c3 0f 1f 40
[4381929.990193] rbd: rbd0: encountered watch error: -107
[4381932.998683] INFO: rcu_sched self-detected stall on CPU { 14} (t=60001 jiffies g=560339035 c=560339034 q=106884641)
[4381932.999717] INFO: rcu_sched detected stalls on CPUs/tasks: { 14} (detected by 15, t=60002 jiffies, g=560339035, c=560339034, q=106884641)
[4381932.999719] Task dump for CPU 14:
[4381932.999726] mount R running task 0 2760216 4131761 0x00000088
[4381932.999728] Call Trace:
[4381932.999742] [<ffffffffac7167f0>] ? _raw_spin_lock+0x10/0x30
[4381932.999747] [<ffffffffac232e7b>] ? dentry_lru_del+0x2b/0x70
[4381932.999751] [<ffffffffac2338d1>] ? __dentry_kill+0x51/0x180
[4381932.999754] [<ffffffffac233cbe>] ? shrink_dentry_list+0xde/0x230
[4381932.999757] [<ffffffffac233eaa>] ? shrink_dcache_sb+0x9a/0xe0
[4381932.999762] [<ffffffffac21fa71>] ? do_remount_sb+0x51/0x200
[4381932.999767] [<ffffffffac240047>] ? do_mount+0x757/0xce0
[4381932.999773] [<ffffffffac1b3b82>] ? memdup_user+0x42/0x70
[4381932.999777] [<ffffffffac240913>] ? SyS_mount+0x83/0xd0
[4381932.999783] [<ffffffffac720795>] ? system_call_fastpath+0x1c/0x21
[4381933.002072] Task dump for CPU 14:
[4381933.002074] mount R running task 0 2760216 4131761 0x00000088
[4381933.002078] Call Trace:
[4381933.002082] <IRQ> [<ffffffffac0ce8c8>] sched_show_task+0xa8/0x110
[4381933.002097] [<ffffffffac0d2499>] dump_cpu_task+0x39/0x70
[4381933.002104] [<ffffffffac148fc0>] rcu_dump_cpu_stacks+0x90/0xd0
[4381933.002108] [<ffffffffac14c662>] rcu_check_callbacks+0x442/0x730
[4381933.002113] [<ffffffffac101c10>] ? tick_sched_do_timer+0x50/0x50
[4381933.002121] [<ffffffffac0a4f46>] update_process_times+0x46/0x80
[4381933.002124] [<ffffffffac101a10>] tick_sched_handle+0x30/0x70
[4381933.002126] [<ffffffffac101c49>] tick_sched_timer+0x39/0x80
[4381933.002131] [<ffffffffac0bf7e6>] __hrtimer_run_queues+0xd6/0x260
[4381933.002134] [<ffffffffac0bfd7f>] hrtimer_interrupt+0xaf/0x1d0
[4381933.002141] [<ffffffffac05847b>] local_apic_timer_interrupt+0x3b/0x60
[4381933.002146] [<ffffffffac725063>] smp_apic_timer_interrupt+0x43/0x60
[4381933.002151] [<ffffffffac7217b2>] apic_timer_interrupt+0x162/0x170
[4381933.002152] <EOI> [<ffffffffac7167f0>] ? _raw_spin_lock+0x10/0x30
[4381933.002159] [<ffffffffac232e7b>] ? dentry_lru_del+0x2b/0x70
[4381933.002162] [<ffffffffac2338d1>] __dentry_kill+0x51/0x180
[4381933.002164] [<ffffffffac233cbe>] shrink_dentry_list+0xde/0x230
[4381933.002166] [<ffffffffac233eaa>] shrink_dcache_sb+0x9a/0xe0
[4381933.002170] [<ffffffffac21fa71>] do_remount_sb+0x51/0x200
[4381933.002174] [<ffffffffac240047>] do_mount+0x757/0xce0
[4381933.002178] [<ffffffffac1b3b82>] ? memdup_user+0x42/0x70
[4381933.002180] [<ffffffffac240913>] SyS_mount+0x83/0xd0
[4381933.002183] [<ffffffffac720795>] system_call_fastpath+0x1c/0x21
[4381959.770291] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [mount:2760216]
[4381959.771426] Modules linked in: vfat fat isofs ip_vs fuse ext4 mbcache jbd2 rbd libceph dns_resolver cfg80211 rfkill udp_diag unix_diag tcp_diag inet_diag veth nf_conntrack_netlink nfnetlink xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_mark xt_comment ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay(T) scsi_transport_iscsi bonding vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_ssif joydev mei_me mei iTCO_wdt iTCO_vendor_support
[4381959.771479] pcspkr dcdbas ipmi_si ipmi_devintf ipmi_msghandler shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod sd_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ixgbe libata crc32c_intel tg3 megaraid_sas i2c_core mdio dca ptp pps_core dm_mirror dm_region_hash dm_log dm_snapshot target_core_user uio target_core_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common dm_multipath dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_mod libcrc32c
[4381959.771521] CPU: 14 PID: 2760216 Comm: mount Kdump: loaded Tainted: G W L ------------ T 3.10.0-862.6.3.el7.x86_64 #1
[4381959.771523] Hardware name: Dell Inc. PowerEdge R720/0DCWD1, BIOS 2.6.1 02/12/2018
[4381959.771526] task: ffff93917178dee0 ti: ffff93a1cf22c000 task.ti: ffff93a1cf22c000
[4381959.771528] RIP: 0010:[<ffffffffac14a158>] [<ffffffffac14a158>] __call_rcu+0x98/0x2c0
[4381959.771540] RSP: 0018:ffff93a1cf22fd30 EFLAGS: 00000246
[4381959.771541] RAX: 0000000009138919 RBX: ffff939c3f9dbb80 RCX: ffffffffacd41e20
[4381959.771543] RDX: ffffffffacc73000 RSI: 0000000000014340 RDI: 0000000000000246
[4381959.771545] RBP: ffff93a1cf22fd58 R08: 0000000000000000 R09: 0000000000000000
[4381959.771546] R10: ffff939c3f9dbb80 R11: ffffdb1b021f3800 R12: 000000002a7c93a8
[4381959.771548] R13: ffff93a1cf22fd58 R14: ffff93a1cf22fcb0 R15: ffff938cc7ce208f
[4381959.771550] FS: 00007fc6c57db880(0000) GS:ffff939c3f9c0000(0000) knlGS:0000000000000000
[4381959.771552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4381959.771554] CR2: 00007fc6c499c15c CR3: 0000001084400000 CR4: 00000000001607e0
[4381959.771556] Call Trace:
[4381959.771563] [<ffffffffac14a39d>] call_rcu_sched+0x1d/0x20
[4381959.771568] [<ffffffffac2333cf>] d_free+0x4f/0x70
[4381959.771570] [<ffffffffac2339ea>] __dentry_kill+0x16a/0x180
[4381959.771573] [<ffffffffac233cbe>] shrink_dentry_list+0xde/0x230
[4381959.771575] [<ffffffffac233eaa>] shrink_dcache_sb+0x9a/0xe0
[4381959.771581] [<ffffffffac21fa71>] do_remount_sb+0x51/0x200
[4381959.771598] [<ffffffffac240047>] do_mount+0x757/0xce0
[4381959.771604] [<ffffffffac1b3b82>] ? memdup_user+0x42/0x70
[4381959.771606] [<ffffffffac240913>] SyS_mount+0x83/0xd0
[4381959.771614] [<ffffffffac720795>] system_call_fastpath+0x1c/0x21
[4381959.771615] Code: 3c cd a0 53 d3 ac 80 3d 66 eb 02 01 00 8b 87 70 01 00 00 0f 85 3a 01 00 00 80 3d 3f ba bc 00 00 0f 84 bd 01 00 00 4c 89 ef 57 9d <0f> 1f 44 00 00 48 83 c4 10 5b 41 5c 41 5d 5d c3 0f 1f 84 00 00
[4381959.995610] rbd: rbd0: encountered watch error: -107
[4381987.769883] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [mount:2760216]
[4381987.771033] Modules linked in: vfat fat isofs ip_vs fuse ext4 mbcache jbd2 rbd libceph dns_resolver cfg80211 rfkill udp_diag unix_diag tcp_diag inet_diag veth nf_conntrack_netlink nfnetlink xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_mark xt_comment ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay(T) scsi_transport_iscsi bonding vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_ssif joydev mei_me mei iTCO_wdt iTCO_vendor_support
[4381987.771086] pcspkr dcdbas ipmi_si ipmi_devintf ipmi_msghandler shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod sd_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ixgbe libata crc32c_intel tg3 megaraid_sas i2c_core mdio dca ptp pps_core dm_mirror dm_region_hash dm_log dm_snapshot target_core_user uio target_core_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common dm_multipath dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_mod libcrc32c
[4381987.771127] CPU: 14 PID: 2760216 Comm: mount Kdump: loaded Tainted: G W L ------------ T 3.10.0-862.6.3.el7.x86_64 #1
[4381987.771129] Hardware name: Dell Inc. PowerEdge R720/0DCWD1, BIOS 2.6.1 02/12/2018
[4381987.771132] task: ffff93917178dee0 ti: ffff93a1cf22c000 task.ti: ffff93a1cf22c000
[4381987.771134] RIP: 0010:[<ffffffffac7167f0>] [<ffffffffac7167f0>] _raw_spin_lock+0x10/0x30
[4381987.771145] RSP: 0018:ffff93a1cf22fdb8 EFLAGS: 00000246
[4381987.771147] RAX: 0000000000000000 RBX: ffffffffffffff10 RCX: 00b083a500000000
[4381987.771148] RDX: 0000000000000001 RSI: 0000000000b083a5 RDI: ffff939ccf1a6958
[4381987.771150] RBP: ffff93a1cf22fdf8 R08: 0000000000000000 R09: 0000000000000000
[4381987.771151] R10: ffff939c3f9dbb80 R11: ffffdb1b021f3800 R12: ffffffffac232ec0
[4381987.771153] R13: ffffffffacc73000 R14: 00000000ffffffff R15: 0000000000000000
[4381987.771155] FS: 00007fc6c57db880(0000) GS:ffff939c3f9c0000(0000) knlGS:0000000000000000
[4381987.771157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4381987.771171] CR2: 00007fc6c499c15c CR3: 0000001084400000 CR4: 00000000001607e0
[4381987.771174] Call Trace:
[4381987.771181] [<ffffffffac233c1c>] ? shrink_dentry_list+0x3c/0x230
[4381987.771185] [<ffffffffac233eaa>] shrink_dcache_sb+0x9a/0xe0
[4381987.771190] [<ffffffffac21fa71>] do_remount_sb+0x51/0x200
[4381987.771195] [<ffffffffac240047>] do_mount+0x757/0xce0
[4381987.771200] [<ffffffffac1b3b82>] ? memdup_user+0x42/0x70
[4381987.771202] [<ffffffffac240913>] SyS_mount+0x83/0xd0
[4381987.771208] [<ffffffffac720795>] system_call_fastpath+0x1c/0x21
[4381987.771210] Code: 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 c5 2c ff ff 5d c3 0f 1f 40
[4381990.001417] rbd: rbd0: encountered watch error: -107
[4382009.590969] libceph: mon0 192.168.100.74:6789 session lost, hunting for new mon
[4382009.593496] libceph: mon2 192.168.100.75:6789 session established
[4382016.237493] ixgbe 0000:05:00.0 p6p1: initiating reset due to tx timeout
[4382016.237555] ixgbe 0000:05:00.0 p6p1: Reset adapter
[4382035.769185] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [etcd:214637]
[4382035.770413] Modules linked in: vfat fat isofs ip_vs fuse ext4 mbcache jbd2 rbd libceph dns_resolver cfg80211 rfkill udp_diag unix_diag tcp_diag inet_diag veth nf_conntrack_netlink nfnetlink xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_mark xt_comment ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat xt_addrtype iptable_filter xt_conntrack br_netfilter bridge stp llc overlay(T) scsi_transport_iscsi bonding vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_ssif joydev mei_me mei iTCO_wdt iTCO_vendor_support
[4382035.770480] pcspkr dcdbas ipmi_si ipmi_devintf ipmi_msghandler shpchp lpc_ich acpi_pad acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod sd_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ixgbe libata crc32c_intel tg3 megaraid_sas i2c_core mdio dca ptp pps_core dm_mirror dm_region_hash dm_log dm_snapshot target_core_user uio target_core_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common dm_multipath dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_mod libcrc32c
[4382035.770521] CPU: 14 PID: 214637 Comm: etcd Kdump: loaded Tainted: G W L ------------ T 3.10.0-862.6.3.el7.x86_64 #1
[4382035.770523] Hardware name: Dell Inc. PowerEdge R720/0DCWD1, BIOS 2.6.1 02/12/2018
[4382035.770526] task: ffff93a7ff2e2f70 ti: ffff93aaefe94000 task.ti: ffff93aaefe94000
[4382035.770528] RIP: 0010:[<ffffffffac232ed8>] [<ffffffffac232ed8>] __d_free+0x18/0x40
[4382035.770536] RSP: 0000:ffff939c3f9c3ea8 EFLAGS: 00000292
[4382035.770538] RAX: ffffffffac232ec0 RBX: ffff93a58a68eb40 RCX: 00000001002a001f
[4382035.770540] RDX: ffff93a9cca75eb0 RSI: ffffdb1b76329d00 RDI: ffff93a9cca75e38
[4382035.770541] RBP: ffff939c3f9c3eb0 R08: ffff93a9cca74b40 R09: 00000001002a001f
[4382035.770543] R10: 00000000cca75501 R11: ffffdb1b76329d00 R12: ffff939c3f9c3e18
[4382035.770544] R13: ffffffffac7217b2 R14: ffff939c3f9c3eb0 R15: ffff93a9cca75e00
[4382035.770546] FS: 00007f36d7fff700(0000) GS:ffff939c3f9c0000(0000) knlGS:0000000000000000
[4382035.770548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4382035.770550] CR2: 000000c431171000 CR3: 0000001d1a2a4000 CR4: 00000000001607e0
[4382035.770552] Call Trace:
[4382035.770555] <IRQ>
[4382035.770565] [<ffffffffac14b2b0>] rcu_process_callbacks+0x1e0/0x580
[4382035.770572] [<ffffffffac09b085>] __do_softirq+0xf5/0x280
[4382035.770577] [<ffffffffac723cec>] call_softirq+0x1c/0x30
[4382035.770582] [<ffffffffac02d625>] do_softirq+0x65/0xa0
[4382035.770585] [<ffffffffac09b405>] irq_exit+0x105/0x110
[4382035.770588] [<ffffffffac725068>] smp_apic_timer_interrupt+0x48/0x60
[4382035.770593] [<ffffffffac7217b2>] apic_timer_interrupt+0x162/0x170
[4382035.770594] <EOI>
[4382035.770596] Code:
[4382035.770597] 00 00 01 c6 07 00 0f 1f 40 00 5b 41 5c 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 53 48 8d 9f 50 ff ff ff 48 8b bf 78 ff ff ff <48> 8d 43 38 48 39 c7 74 05 e8 ba 43 fc ff 48 8b 3d 4b 65 b1 00
[4382048.912110] libceph: mon2 192.168.100.75:6789 session lost, hunting for new mon
[4382048.921812] rbd: rbd0: encountered watch error: -107
[4382049.175254] bond2: link status definitely down for interface p6p1, disabling it
[4382049.243019] ixgbe 0000:05:00.0 p6p1: detected SFP+: 5_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
