Hi! Thank you. That's how clister.conf looks like: --- 8< --- <?xml version="1.0"?> <cluster name="ckvm1_pod1" config_version="5"> <clusternodes> <clusternode name="bond0-101.host1.ckvm1.pod1.***.net" votes="1" nodeid="1"><fence><method name="single"><device name="host1_ipmi"/></method></fence></clusternode><clusternode name="bond0-101.host2.ckvm1.pod1.***.net" votes="1" nodeid="2"><fence><method name="single"><device name="host2_ipmi"/></method></fence></clusternode><clusternode name="bond0-101.host3.ckvm1.pod1.***.net" votes="1" nodeid="3"><fence><method name="single"><device name="host3_ipmi"/></method></fence></clusternode><clusternode name="bond0-101.host4.ckvm1.pod1.***.net" votes="1" nodeid="4"><fence><method name="single"><device name="host4_ipmi"/></method></fence></clusternode></clusternodes> <fencedevices> <fencedevice name="host1_ipmi" agent="fence_ipmilan" ipaddr="mc.host1.ckvm1.pod1.***.net" login="***" passwd="***"/><fencedevice name="host2_ipmi" agent="fence_ipmilan" ipaddr="mc.host2.ckvm1.pod1.***.net" login="***" passwd="***"/>="***"/><fencedevice name="host3_ipmi" agent="fence_ipmilan" ipaddr="mc.host3.ckvm1.pod1.***.net" login="***" passwd="***"/>="***"/><fencedevice name="host4_ipmi" agent="fence_ipmilan" ipaddr="mc.host4.ckvm1.pod1.***.net" login="***" passwd="***"/></fencedevices> <rm> <failoverdomains/> <resources/> </rm> </cluster> --- >8 --- That's what I've got in logg on the affected node: --- 8< --- Oct 24 13:01:57 host1 kernel: sd 10:0:0:0: [sdb] Synchronizing SCSI cache Oct 24 13:01:57 host1 kernel: connection6:0: detected conn error (1020) Oct 24 13:01:58 host1 kernel: scsi11 : iSCSI Initiator over TCP/IP Oct 24 13:01:58 host1 kernel: scsi 11:0:0:0: Direct-Access OSNEXUS QUANTASTOR 300 PQ: 0 ANSI: 6 Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 3478923509 512-byte logical blocks: (1.78 TB/1.61 TiB) Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 4096-byte physical blocks Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write Protect is off Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: Attached scsi generic sg2 type 0 Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Oct 24 13:01:58 host1 kernel: sdc: unknown partition table Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Attached SCSI disk Oct 24 13:01:58 host1 iscsid: Connection6:0 to [target: iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal: 192.168.***.***,3260] through [iface: default] is shutdown. Oct 24 13:01:58 host1 iscsid: Connection7:0 to [target: iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal: 192.168.***.***,3260] through [iface: default] is operational now Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Not tainted) Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_i pv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_ tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO _vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Not tainted 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:01 host1 kernel: Call Trace: Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:01 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 Oct 24 13:02:01 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:01 host1 kernel: [<ffffffff81083e26>] ? group_send_sig_info+0x56/0x70 Oct 24 13:02:01 host1 kernel: [<ffffffff8100bc6e>] ? invalidate_interrupt1+0xe/0x20 Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:01 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 Oct 24 13:02:01 host1 kernel: [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0 Oct 24 13:02:01 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6447 ]--- Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:01 host1 kernel: Call Trace: Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:01 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 Oct 24 13:02:01 host1 kernel: [<ffffffff81115a40>] ? generic_file_aio_read+0x0/0x700 Oct 24 13:02:01 host1 kernel: [<ffffffff8117ad5b>] ? do_sync_readv_writev+0xfb/0x140 Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:01 host1 kernel: [<ffffffff8117be06>] ? do_readv_writev+0xd6/0x1f0 Oct 24 13:02:01 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 Oct 24 13:02:01 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6448 ]--- Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:02 host1 kernel: Call Trace: Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:02 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 Oct 24 13:02:02 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffff81083e26>] ? group_send_sig_info+0x56/0x70 Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:02 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 Oct 24 13:02:02 host1 kernel: [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0 Oct 24 13:02:02 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d6449 ]--- Oct 24 13:02:02 host1 abrt-dump-oops: Reported 3 kernel oopses to Abrt Oct 24 13:02:02 host1 abrtd: Directory 'oops-2013-10-24-13:02:02-2922-2' creation detected Oct 24 13:02:02 host1 abrtd: Can't open file '/var/spool/abrt/oops-2013-10-24-13:02:02-2922-2/uid': No such file or directory Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:02 host1 kernel: Call Trace: Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:02 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:02 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 Oct 24 13:02:02 host1 kernel: [<ffffffff81115a40>] ? generic_file_aio_read+0x0/0x700 Oct 24 13:02:02 host1 kernel: [<ffffffff8117ad5b>] ? do_sync_readv_writev+0xfb/0x140 Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:02 host1 kernel: [<ffffffff8117be06>] ? do_readv_writev+0xd6/0x1f0 Oct 24 13:02:02 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 Oct 24 13:02:02 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d644a ]--- Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:03 host1 kernel: Call Trace: Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:03 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 Oct 24 13:02:03 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffff81083e26>] ? group_send_sig_info+0x56/0x70 Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:03 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 Oct 24 13:02:03 host1 kernel: [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0 Oct 24 13:02:03 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644b ]--- Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:03 host1 kernel: Call Trace: Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:03 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:03 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 Oct 24 13:02:03 host1 kernel: [<ffffffff81115a40>] ? generic_file_aio_read+0x0/0x700 Oct 24 13:02:03 host1 kernel: [<ffffffff8117ad5b>] ? do_sync_readv_writev+0xfb/0x140 Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:03 host1 kernel: [<ffffffff8117be06>] ? do_readv_writev+0xd6/0x1f0 Oct 24 13:02:03 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 Oct 24 13:02:03 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644c ]--- Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:04 host1 kernel: Call Trace: Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:04 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 Oct 24 13:02:04 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffff81083e26>] ? group_send_sig_info+0x56/0x70 Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:04 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 Oct 24 13:02:04 host1 kernel: [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0 Oct 24 13:02:04 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644d ]--- Oct 24 13:02:04 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:04 host1 kernel: Call Trace: Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:04 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:04 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 Oct 24 13:02:04 host1 kernel: [<ffffffff81115a40>] ? generic_file_aio_read+0x0/0x700 Oct 24 13:02:04 host1 kernel: [<ffffffff8117ad5b>] ? do_sync_readv_writev+0xfb/0x140 Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:04 host1 kernel: [<ffffffff8117be06>] ? do_readv_writev+0xd6/0x1f0 Oct 24 13:02:04 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 Oct 24 13:02:04 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644e ]--- Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 434865136 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 434865136 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 434865150 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 434865150 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 0 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 0 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 1 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 434865151 Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block 1 Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:05 host1 kernel: Call Trace: Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ? group_send_sig_info+0x56/0x70 Oct 24 13:02:05 host1 kernel: [<ffffffff81083dd0>] ? group_send_sig_info+0x0/0x70 Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0 Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d644f ]--- Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:05 host1 kernel: Call Trace: Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:05 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ? pagevec_lookup_tag+0x25/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 __brelse+0x3a/0x40() (Tainted: G W --------------- ) Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs] Oct 24 13:02:05 host1 kernel: Pid: 3864, comm: qemu-kvm Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:05 host1 kernel: Call Trace: Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0 Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? warn_slowpath_fmt+0x46/0x50 Oct 24 13:02:05 host1 kernel: [<ffffffff81116a25>] ? mempool_free+0x95/0xa0 Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? gfs2_block_map+0x1e1/0xf30 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? gup_pte_range+0xe5/0x130 Oct 24 13:02:05 host1 kernel: [<ffffffff81115a40>] ? generic_file_aio_read+0x0/0x700 Oct 24 13:02:05 host1 kernel: [<ffffffff8117ad5b>] ? do_sync_readv_writev+0xfb/0x140 Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? gup_pud_range+0x148/0x1d0 Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? gfs2_get_block_direct+0x10/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? __blockdev_direct_IO_newtrunc+0x85a/0xb90 Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? __blockdev_direct_IO+0x5e/0xd0 Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:05 host1 kernel: [<ffffffff8117be06>] ? do_readv_writev+0xd6/0x1f0 Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? gfs2_direct_IO+0x100/0x110 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 Oct 24 13:02:05 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? gfs2_get_block_direct+0x0/0x20 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6450 ]--- Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? gfs2_direct_IO+0x5c/0x110 [gfs2] Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? generic_file_aio_read+0x6bb/0x700 Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ? group_send_sig_info+0x56/0x70 Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? selinux_file_permission+0xfb/0x150 Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? security_file_permission+0x16/0x20 Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ? audit_syscall_entry+0x272/0x2a0 Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6451 ]--- Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: fatal: I/O error Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: block = 14244 Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: function = log_write_header, file = fs/gfs2/log.c, line = 616 Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: about to withdraw this file system Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: telling LM to unmount Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: withdrawn Oct 24 13:02:06 host1 kernel: Pid: 4029, comm: parted Tainted: G W --------------- 2.6.32-279.5.1.el6.x86_64 #1 Oct 24 13:02:06 host1 kernel: Call Trace: Oct 24 13:02:06 host1 kernel: [<ffffffffa07b6062>] ? gfs2_lm_withdraw+0x102/0x130 [gfs2] Oct 24 13:02:06 host1 kernel: [<ffffffff814feb58>] ? out_of_line_wait_on_bit+0x78/0x90 Oct 24 13:02:06 host1 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50 Oct 24 13:02:06 host1 kernel: [<ffffffffa07b60d0>] ? gfs2_io_error_bh_i+0x40/0x50 [gfs2] Oct 24 13:02:06 host1 kernel: [<ffffffff811ae0b6>] ? __wait_on_buffer+0x26/0x30 Oct 24 13:02:06 host1 kernel: [<ffffffffa079e288>] ? log_write_header+0x3a8/0x490 [gfs2] Oct 24 13:02:06 host1 kernel: [<ffffffffa079e951>] ? gfs2_log_flush+0x301/0x6f0 [gfs2] Oct 24 13:02:06 host1 kernel: [<ffffffff811a48a9>] ? sync_inodes_sb+0x149/0x190 Oct 24 13:02:06 host1 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Oct 24 13:02:06 host1 kernel: [<ffffffffa07b24cc>] ? gfs2_sync_fs+0x2c/0x30 [gfs2] Oct 24 13:02:06 host1 kernel: [<ffffffff811aa2e3>] ? __sync_filesystem+0x53/0x90 Oct 24 13:02:06 host1 kernel: [<ffffffff811aa51b>] ? sync_filesystem+0x4b/0x70 Oct 24 13:02:06 host1 kernel: [<ffffffff811b514e>] ? fsync_bdev+0x2e/0x70 Oct 24 13:02:06 host1 kernel: [<ffffffff8125ea9b>] ? blkdev_ioctl+0x3ab/0x6e0 Oct 24 13:02:06 host1 kernel: [<ffffffff811b391c>] ? block_ioctl+0x3c/0x40 Oct 24 13:02:06 host1 kernel: [<ffffffff8118dff2>] ? vfs_ioctl+0x22/0xa0 Oct 24 13:02:06 host1 kernel: [<ffffffff8118e194>] ? do_vfs_ioctl+0x84/0x580 Oct 24 13:02:06 host1 kernel: [<ffffffff8118e711>] ? sys_ioctl+0x81/0xa0 Oct 24 13:02:06 host1 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Oct 24 13:02:06 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt Oct 24 13:02:29 host1 kernel: __ratelimit: 562 callbacks suppressed Oct 24 13:02:29 host1 kernel: kvm: 23115: cpu0 unhandled wrmsr: 0x198 data 0 --- >8 --- Now I have this "uncontrolled entry" and it doesn't allow me to start cman again: --- 8< --- Oct 24 18:07:42 host1 dlm_controld[16712]: dlm_controld 3.0.12.1 started Oct 24 18:07:43 host1 gfs_controld[16761]: gfs_controld 3.0.12.1 started Oct 24 18:07:43 host1 gfs_controld[16761]: found uncontrolled gfs2 fs ckvm1_pod1:vg_psv4 Oct 24 18:07:43 host1 gfs_controld[16761]: telling cman to remove nodeid 1 from cluster Oct 24 18:07:43 host1 corosync[16628]: cman killed by node 1 because we were killed by cman_tool or other application Oct 24 18:07:43 host1 dlm_controld[16712]: cluster is down, exiting Oct 24 18:07:43 host1 dlm_controld[16712]: daemon cpg_dispatch error 2 --- >8 --- How to get rid of it? How to make node to "forget" about this gfs2? Thank you! -----Original Message----- From: linux-cluster-bounces@xxxxxxxxxx [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Digimer Sent: Thursday, October 24, 2013 5:20 PM To: linux clustering Subject: Re: uncontrolled entry On 24/10/13 10:18, Vladimir Melnik wrote: > Dear colleagues, > > I'll be very grateful for any clue or advice. I have a cluster > (GFS2/DLM on iSCSI). Something happened today, so one node has stopped > working in cluster and all operations with the filesystem was giving "input/output errors". > Other nodes are working okay. > > I unmounted the filesystem and stopped all services (gfs2, clvmd, > cman, iscsi, iscsid), but I can't start cman again. That's what I have > in > log-files: > fenced[2801]: found uncontrolled entry > /sys/fs/gfs2/ckvm1_pod1:vg_psv4 > gfs_controld[2878]: found uncontrolled gfs2 fs ckvm1_pod1:vg_psv4 > > Yes, I really have this "uncontrolled entry", how to get rid of it > without rebooting the node? > > Thank you very much for any ideas and hints. I've not seen that particular error before. Can you paste your cluster.conf and the log files from the effected node, starting just before the event occured? Please also look at the logs from other node(s) at the same time and paste how they responded to the node failure. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster