That was extremely hard to read, and the cluster.conf was edited to such a degree that it was no longer formatted correctly. That said, I think I can see enough to have an idea what happened. You do not manage the iSCSI connection or the gfs2 partition in the cluster, so the failure of the iSCSI connection and subsequent failure of the gfs2 partition were not a concern of the cluster. Corosync on that node was not interrupted, so the node was not fenced (that's only needed to put a node in an unknown state back into a known state). Have you resolved the iSCSI connection problem? As for gfs2, you stopped the cluster while it was still (trying to) run, so it's probably blocked. I'm not a GFS2 expert, so I can't speak to the internals of what is going on. If it was me, I would reboot the node, make sure iscsi was working after the reboot and then restart cman. I'd also put the storage components under the cluster's control, too. digimer On 24/10/13 11:11, Vladimir Melnik wrote: > Hi! > > Thank you. That's how clister.conf looks like: > > --- 8< --- > <?xml version="1.0"?> > <cluster name="ckvm1_pod1" config_version="5"> > <clusternodes> > <clusternode name="bond0-101.host1.ckvm1.pod1.***.net" votes="1" > nodeid="1"><fence><method name="single"><device > name="host1_ipmi"/></method></fence></clusternode><clusternode > name="bond0-101.host2.ckvm1.pod1.***.net" votes="1" > nodeid="2"><fence><method name="single"><device > name="host2_ipmi"/></method></fence></clusternode><clusternode > name="bond0-101.host3.ckvm1.pod1.***.net" votes="1" > nodeid="3"><fence><method name="single"><device > name="host3_ipmi"/></method></fence></clusternode><clusternode > name="bond0-101.host4.ckvm1.pod1.***.net" votes="1" > nodeid="4"><fence><method name="single"><device > name="host4_ipmi"/></method></fence></clusternode></clusternodes> > <fencedevices> > <fencedevice name="host1_ipmi" agent="fence_ipmilan" > ipaddr="mc.host1.ckvm1.pod1.***.net" login="***" passwd="***"/><fencedevice > name="host2_ipmi" agent="fence_ipmilan" ipaddr="mc.host2.ckvm1.pod1.***.net" > login="***" passwd="***"/>="***"/><fencedevice name="host3_ipmi" > agent="fence_ipmilan" ipaddr="mc.host3.ckvm1.pod1.***.net" login="***" > passwd="***"/>="***"/><fencedevice name="host4_ipmi" agent="fence_ipmilan" > ipaddr="mc.host4.ckvm1.pod1.***.net" login="***" > passwd="***"/></fencedevices> > <rm> > <failoverdomains/> > <resources/> > </rm> > </cluster> > --- >8 --- > > That's what I've got in logg on the affected node: > --- 8< --- > Oct 24 13:01:57 host1 kernel: sd 10:0:0:0: [sdb] Synchronizing SCSI cache > Oct 24 13:01:57 host1 kernel: connection6:0: detected conn error (1020) > Oct 24 13:01:58 host1 kernel: scsi11 : iSCSI Initiator over TCP/IP > Oct 24 13:01:58 host1 kernel: scsi 11:0:0:0: Direct-Access OSNEXUS > QUANTASTOR 300 PQ: 0 ANSI: 6 > Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 3478923509 512-byte logical > blocks: (1.78 TB/1.61 TiB) > Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 4096-byte physical blocks > Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write Protect is off > Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: Attached scsi generic sg2 type 0 > Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write cache: enabled, read > cache: enabled, supports DPO and FUA > Oct 24 13:01:58 host1 kernel: sdc: unknown partition table > Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Attached SCSI disk > Oct 24 13:01:58 host1 iscsid: Connection6:0 to [target: > iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal: > 192.168.***.***,3260] through [iface: default] is shutdown. > Oct 24 13:01:58 host1 iscsid: Connection7:0 to [target: > iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal: > 192.168.***.***,3260] through [iface: default] is operational now > Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Not tainted) > Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss > exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf > bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT > iptable_filter iptable_nat nf_nat nf_conntrack_i > pv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i > cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm > ib_sa ib_mad ib_core ib_addr ipv6 iscsi_ > tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO > _vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 > sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror > dm_region_hash dm_log dm_mod [last unloaded: > configfs] > Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Not tainted > 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:01 host1 kernel: Call Trace: > Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:01 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 > Oct 24 13:02:01 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 > Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:01 host1 kernel: [<ffffffff81083e26>] ? > group_send_sig_info+0x56/0x70 > Oct 24 13:02:01 host1 kernel: [<ffffffff8100bc6e>] ? > invalidate_interrupt1+0xe/0x20 > Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:01 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 > Oct 24 13:02:01 host1 kernel: [<ffffffff810d6b12>] ? > audit_syscall_entry+0x272/0x2a0 > Oct 24 13:02:01 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 > Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6447 ]--- > Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:01 host1 kernel: Call Trace: > Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:01 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 > Oct 24 13:02:01 host1 kernel: [<ffffffff81115a40>] ? > generic_file_aio_read+0x0/0x700 > Oct 24 13:02:01 host1 kernel: [<ffffffff8117ad5b>] ? > do_sync_readv_writev+0xfb/0x140 > Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:01 host1 kernel: [<ffffffff8117be06>] ? > do_readv_writev+0xd6/0x1f0 > Oct 24 13:02:01 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 > Oct 24 13:02:01 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 > Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6448 ]--- > Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:02 host1 kernel: Call Trace: > Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:02 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 > Oct 24 13:02:02 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 > Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffff81083e26>] ? > group_send_sig_info+0x56/0x70 > Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:02 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 > Oct 24 13:02:02 host1 kernel: [<ffffffff810d6b12>] ? > audit_syscall_entry+0x272/0x2a0 > Oct 24 13:02:02 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 > Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d6449 ]--- > Oct 24 13:02:02 host1 abrt-dump-oops: Reported 3 kernel oopses to Abrt > Oct 24 13:02:02 host1 abrtd: Directory 'oops-2013-10-24-13:02:02-2922-2' > creation detected > Oct 24 13:02:02 host1 abrtd: Can't open file > '/var/spool/abrt/oops-2013-10-24-13:02:02-2922-2/uid': No such file or > directory > Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:02 host1 kernel: Call Trace: > Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:02 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:02 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 > Oct 24 13:02:02 host1 kernel: [<ffffffff81115a40>] ? > generic_file_aio_read+0x0/0x700 > Oct 24 13:02:02 host1 kernel: [<ffffffff8117ad5b>] ? > do_sync_readv_writev+0xfb/0x140 > Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:02 host1 kernel: [<ffffffff8117be06>] ? > do_readv_writev+0xd6/0x1f0 > Oct 24 13:02:02 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 > Oct 24 13:02:02 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 > Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d644a ]--- > Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:03 host1 kernel: Call Trace: > Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:03 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 > Oct 24 13:02:03 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 > Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffff81083e26>] ? > group_send_sig_info+0x56/0x70 > Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:03 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 > Oct 24 13:02:03 host1 kernel: [<ffffffff810d6b12>] ? > audit_syscall_entry+0x272/0x2a0 > Oct 24 13:02:03 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 > Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644b ]--- > Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:03 host1 kernel: Call Trace: > Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:03 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:03 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 > Oct 24 13:02:03 host1 kernel: [<ffffffff81115a40>] ? > generic_file_aio_read+0x0/0x700 > Oct 24 13:02:03 host1 kernel: [<ffffffff8117ad5b>] ? > do_sync_readv_writev+0xfb/0x140 > Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:03 host1 kernel: [<ffffffff8117be06>] ? > do_readv_writev+0xd6/0x1f0 > Oct 24 13:02:03 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 > Oct 24 13:02:03 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 > Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644c ]--- > Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:04 host1 kernel: Call Trace: > Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:04 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 > Oct 24 13:02:04 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 > Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffff81083e26>] ? > group_send_sig_info+0x56/0x70 > Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:04 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 > Oct 24 13:02:04 host1 kernel: [<ffffffff810d6b12>] ? > audit_syscall_entry+0x272/0x2a0 > Oct 24 13:02:04 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 > Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644d ]--- > Oct 24 13:02:04 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt > Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:04 host1 kernel: Call Trace: > Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:04 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:04 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 > Oct 24 13:02:04 host1 kernel: [<ffffffff81115a40>] ? > generic_file_aio_read+0x0/0x700 > Oct 24 13:02:04 host1 kernel: [<ffffffff8117ad5b>] ? > do_sync_readv_writev+0xfb/0x140 > Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:04 host1 kernel: [<ffffffff8117be06>] ? > do_readv_writev+0xd6/0x1f0 > Oct 24 13:02:04 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 > Oct 24 13:02:04 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 > Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644e ]--- > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 434865136 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 434865136 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 434865150 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 434865150 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 0 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 0 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 1 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 434865151 > Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical block > 1 > Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:05 host1 kernel: Call Trace: > Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 > Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 > Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ? > group_send_sig_info+0x56/0x70 > Oct 24 13:02:05 host1 kernel: [<ffffffff81083dd0>] ? > group_send_sig_info+0x0/0x70 > Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 > Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ? > audit_syscall_entry+0x272/0x2a0 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 > Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d644f ]--- > Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:05 host1 kernel: Call Trace: > Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:05 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ? > pagevec_lookup_tag+0x25/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ > Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 > __brelse+0x3a/0x40() (Tainted: G W --------------- ) > Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- > Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer > Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 jbd > nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE > xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc > cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp stp > llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat nf_nat > nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi > iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio > ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp > libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun > kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode > serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca > i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi > ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod > [last unloaded: configfs] > Oct 24 13:02:05 host1 kernel: Pid: 3864, comm: qemu-kvm Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:05 host1 kernel: Call Trace: > Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? > warn_slowpath_common+0x87/0xc0 > Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? > warn_slowpath_fmt+0x46/0x50 > Oct 24 13:02:05 host1 kernel: [<ffffffff81116a25>] ? mempool_free+0x95/0xa0 > Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 > Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? > gfs2_block_map+0x1e1/0xf30 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? > gup_pte_range+0xe5/0x130 > Oct 24 13:02:05 host1 kernel: [<ffffffff81115a40>] ? > generic_file_aio_read+0x0/0x700 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117ad5b>] ? > do_sync_readv_writev+0xfb/0x140 > Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? > gup_pud_range+0x148/0x1d0 > Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? > gfs2_get_block_direct+0x10/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? > __blockdev_direct_IO_newtrunc+0x85a/0xb90 > Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? do_promote+0x208/0x330 > [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? > __blockdev_direct_IO+0x5e/0xd0 > Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117be06>] ? > do_readv_writev+0xd6/0x1f0 > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? > gfs2_direct_IO+0x100/0x110 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? > gfs2_get_block_direct+0x0/0x20 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? futex_wait+0x227/0x380 > Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6450 ]--- > Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? > gfs2_direct_IO+0x5c/0x110 [gfs2] > Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? > generic_file_aio_read+0x6bb/0x700 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? do_sync_read+0xfa/0x140 > Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 > Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ? > group_send_sig_info+0x56/0x70 > Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? > selinux_file_permission+0xfb/0x150 > Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? > security_file_permission+0x16/0x20 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 > Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ? > audit_syscall_entry+0x272/0x2a0 > Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 > Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6451 ]--- > Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: fatal: I/O > error > Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: block = > 14244 > Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: function = > log_write_header, file = fs/gfs2/log.c, line = 616 > Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: about to > withdraw this file system > Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: telling LM to > unmount > Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: withdrawn > Oct 24 13:02:06 host1 kernel: Pid: 4029, comm: parted Tainted: G W > --------------- 2.6.32-279.5.1.el6.x86_64 #1 > Oct 24 13:02:06 host1 kernel: Call Trace: > Oct 24 13:02:06 host1 kernel: [<ffffffffa07b6062>] ? > gfs2_lm_withdraw+0x102/0x130 [gfs2] > Oct 24 13:02:06 host1 kernel: [<ffffffff814feb58>] ? > out_of_line_wait_on_bit+0x78/0x90 > Oct 24 13:02:06 host1 kernel: [<ffffffff81092110>] ? > wake_bit_function+0x0/0x50 > Oct 24 13:02:06 host1 kernel: [<ffffffffa07b60d0>] ? > gfs2_io_error_bh_i+0x40/0x50 [gfs2] > Oct 24 13:02:06 host1 kernel: [<ffffffff811ae0b6>] ? > __wait_on_buffer+0x26/0x30 > Oct 24 13:02:06 host1 kernel: [<ffffffffa079e288>] ? > log_write_header+0x3a8/0x490 [gfs2] > Oct 24 13:02:06 host1 kernel: [<ffffffffa079e951>] ? > gfs2_log_flush+0x301/0x6f0 [gfs2] > Oct 24 13:02:06 host1 kernel: [<ffffffff811a48a9>] ? > sync_inodes_sb+0x149/0x190 > Oct 24 13:02:06 host1 kernel: [<ffffffff810920d0>] ? > autoremove_wake_function+0x0/0x40 > Oct 24 13:02:06 host1 kernel: [<ffffffffa07b24cc>] ? gfs2_sync_fs+0x2c/0x30 > [gfs2] > Oct 24 13:02:06 host1 kernel: [<ffffffff811aa2e3>] ? > __sync_filesystem+0x53/0x90 > Oct 24 13:02:06 host1 kernel: [<ffffffff811aa51b>] ? > sync_filesystem+0x4b/0x70 > Oct 24 13:02:06 host1 kernel: [<ffffffff811b514e>] ? fsync_bdev+0x2e/0x70 > Oct 24 13:02:06 host1 kernel: [<ffffffff8125ea9b>] ? > blkdev_ioctl+0x3ab/0x6e0 > Oct 24 13:02:06 host1 kernel: [<ffffffff811b391c>] ? block_ioctl+0x3c/0x40 > Oct 24 13:02:06 host1 kernel: [<ffffffff8118dff2>] ? vfs_ioctl+0x22/0xa0 > Oct 24 13:02:06 host1 kernel: [<ffffffff8118e194>] ? do_vfs_ioctl+0x84/0x580 > Oct 24 13:02:06 host1 kernel: [<ffffffff8118e711>] ? sys_ioctl+0x81/0xa0 > Oct 24 13:02:06 host1 kernel: [<ffffffff8100b0f2>] ? > system_call_fastpath+0x16/0x1b > Oct 24 13:02:06 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt > Oct 24 13:02:29 host1 kernel: __ratelimit: 562 callbacks suppressed > Oct 24 13:02:29 host1 kernel: kvm: 23115: cpu0 unhandled wrmsr: 0x198 data 0 > --- >8 --- > > Now I have this "uncontrolled entry" and it doesn't allow me to start cman > again: > --- 8< --- > Oct 24 18:07:42 host1 dlm_controld[16712]: dlm_controld 3.0.12.1 started > Oct 24 18:07:43 host1 gfs_controld[16761]: gfs_controld 3.0.12.1 started > Oct 24 18:07:43 host1 gfs_controld[16761]: found uncontrolled gfs2 fs > ckvm1_pod1:vg_psv4 > Oct 24 18:07:43 host1 gfs_controld[16761]: telling cman to remove nodeid 1 > from cluster > Oct 24 18:07:43 host1 corosync[16628]: cman killed by node 1 because we were > killed by cman_tool or other application > Oct 24 18:07:43 host1 dlm_controld[16712]: cluster is down, exiting > Oct 24 18:07:43 host1 dlm_controld[16712]: daemon cpg_dispatch error 2 > --- >8 --- > > How to get rid of it? How to make node to "forget" about this gfs2? > > Thank you! > > -----Original Message----- > From: linux-cluster-bounces@xxxxxxxxxx > [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Digimer > Sent: Thursday, October 24, 2013 5:20 PM > To: linux clustering > Subject: Re: uncontrolled entry > > On 24/10/13 10:18, Vladimir Melnik wrote: >> Dear colleagues, >> >> I'll be very grateful for any clue or advice. I have a cluster >> (GFS2/DLM on iSCSI). Something happened today, so one node has stopped >> working in cluster and all operations with the filesystem was giving > "input/output errors". >> Other nodes are working okay. >> >> I unmounted the filesystem and stopped all services (gfs2, clvmd, >> cman, iscsi, iscsid), but I can't start cman again. That's what I have >> in >> log-files: >> fenced[2801]: found uncontrolled entry >> /sys/fs/gfs2/ckvm1_pod1:vg_psv4 >> gfs_controld[2878]: found uncontrolled gfs2 fs ckvm1_pod1:vg_psv4 >> >> Yes, I really have this "uncontrolled entry", how to get rid of it >> without rebooting the node? >> >> Thank you very much for any ideas and hints. > > I've not seen that particular error before. Can you paste your cluster.conf > and the log files from the effected node, starting just before the event > occured? Please also look at the logs from other > node(s) at the same time and paste how they responded to the node failure. > > -- > Digimer > Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is > trapped in the mind of a person without access to education? > > -- > Linux-cluster mailing list > Linux-cluster@xxxxxxxxxx > https://www.redhat.com/mailman/listinfo/linux-cluster > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
