Rebooting the one node should not impact GFS2 on the other nodes. I don't know if you can kill -9 gfs2. In my experience, you need to reboot. There is a decent chance it will fail to shut down, so I usually handle cases like this by going to a good node and running 'fence_node <victim>'. That will forcibly reboot it and tell the cluster to reap any locks the hung node might still have. Of course, I know little about your setup. If you need more authoritative advice, I'd recommend opening a support ticket with Red Hat. On 24/10/13 13:16, Vladimir Melnik wrote: > Thanks a lot! Please, excuse me for ugly formatting. > > iSCSI works, but I'd like to restart GFS2 without rebooting the node as it > works with other storages and it wouldn't be good to interrupt these > processes. > > Any ideas on how to do that? Thank you. > > > -----Original Message----- > From: linux-cluster-bounces@xxxxxxxxxx > [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Digimer > Sent: Thursday, October 24, 2013 6:35 PM > To: linux clustering > Subject: Re: uncontrolled entry > > That was extremely hard to read, and the cluster.conf was edited to such > a degree that it was no longer formatted correctly. That said, I think I > can see enough to have an idea what happened. > > You do not manage the iSCSI connection or the gfs2 partition in the > cluster, so the failure of the iSCSI connection and subsequent failure > of the gfs2 partition were not a concern of the cluster. Corosync on > that node was not interrupted, so the node was not fenced (that's only > needed to put a node in an unknown state back into a known state). > > Have you resolved the iSCSI connection problem? As for gfs2, you stopped > the cluster while it was still (trying to) run, so it's probably > blocked. I'm not a GFS2 expert, so I can't speak to the internals of > what is going on. > > If it was me, I would reboot the node, make sure iscsi was working after > the reboot and then restart cman. I'd also put the storage components > under the cluster's control, too. > > digimer > > On 24/10/13 11:11, Vladimir Melnik wrote: >> Hi! >> >> Thank you. That's how clister.conf looks like: >> >> --- 8< --- >> <?xml version="1.0"?> >> <cluster name="ckvm1_pod1" config_version="5"> >> <clusternodes> >> <clusternode name="bond0-101.host1.ckvm1.pod1.***.net" votes="1" >> nodeid="1"><fence><method name="single"><device >> name="host1_ipmi"/></method></fence></clusternode><clusternode >> name="bond0-101.host2.ckvm1.pod1.***.net" votes="1" >> nodeid="2"><fence><method name="single"><device >> name="host2_ipmi"/></method></fence></clusternode><clusternode >> name="bond0-101.host3.ckvm1.pod1.***.net" votes="1" >> nodeid="3"><fence><method name="single"><device >> name="host3_ipmi"/></method></fence></clusternode><clusternode >> name="bond0-101.host4.ckvm1.pod1.***.net" votes="1" >> nodeid="4"><fence><method name="single"><device >> name="host4_ipmi"/></method></fence></clusternode></clusternodes> >> <fencedevices> >> <fencedevice name="host1_ipmi" agent="fence_ipmilan" >> ipaddr="mc.host1.ckvm1.pod1.***.net" login="***" > passwd="***"/><fencedevice >> name="host2_ipmi" agent="fence_ipmilan" > ipaddr="mc.host2.ckvm1.pod1.***.net" >> login="***" passwd="***"/>="***"/><fencedevice name="host3_ipmi" >> agent="fence_ipmilan" ipaddr="mc.host3.ckvm1.pod1.***.net" login="***" >> passwd="***"/>="***"/><fencedevice name="host4_ipmi" agent="fence_ipmilan" >> ipaddr="mc.host4.ckvm1.pod1.***.net" login="***" >> passwd="***"/></fencedevices> >> <rm> >> <failoverdomains/> >> <resources/> >> </rm> >> </cluster> >> --- >8 --- >> >> That's what I've got in logg on the affected node: >> --- 8< --- >> Oct 24 13:01:57 host1 kernel: sd 10:0:0:0: [sdb] Synchronizing SCSI cache >> Oct 24 13:01:57 host1 kernel: connection6:0: detected conn error (1020) >> Oct 24 13:01:58 host1 kernel: scsi11 : iSCSI Initiator over TCP/IP >> Oct 24 13:01:58 host1 kernel: scsi 11:0:0:0: Direct-Access OSNEXUS >> QUANTASTOR 300 PQ: 0 ANSI: 6 >> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 3478923509 512-byte > logical >> blocks: (1.78 TB/1.61 TiB) >> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] 4096-byte physical blocks >> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write Protect is off >> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: Attached scsi generic sg2 type > 0 >> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Write cache: enabled, > read >> cache: enabled, supports DPO and FUA >> Oct 24 13:01:58 host1 kernel: sdc: unknown partition table >> Oct 24 13:01:58 host1 kernel: sd 11:0:0:0: [sdc] Attached SCSI disk >> Oct 24 13:01:58 host1 iscsid: Connection6:0 to [target: >> iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal: >> 192.168.***.***,3260] through [iface: default] is shutdown. >> Oct 24 13:01:58 host1 iscsid: Connection7:0 to [target: >> iqn.2009-10.com.osnexus:4eb54cf4-3f64da4ba5413ecc:PSV4, portal: >> 192.168.***.***,3260] through [iface: default] is operational now >> Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Not tainted) >> Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss >> exportfs autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf >> bridge bonding 8021q garp stp llc xt_CHECKSUM iptable_mangle ipt_REJECT >> iptable_filter iptable_nat nf_nat nf_conntrack_i >> pv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi iscsi_boot_sysfs bnx2i >> cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm > iw_cm >> ib_sa ib_mad ib_core ib_addr ipv6 iscsi_ >> tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan > tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO >> _vendor_support ioatdma dca i7core_edac edac_core shpchp ext4 mbcache jbd2 >> sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror >> dm_region_hash dm_log dm_mod [last unloaded: >> configfs] >> Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Not tainted >> 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:01 host1 kernel: Call Trace: >> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8117afda>] ? > do_sync_read+0xfa/0x140 >> Oct 24 13:02:01 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 >> Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:01 host1 kernel: [<ffffffff81083e26>] ? >> group_send_sig_info+0x56/0x70 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8100bc6e>] ? >> invalidate_interrupt1+0xe/0x20 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 >> Oct 24 13:02:01 host1 kernel: [<ffffffff810d6b12>] ? >> audit_syscall_entry+0x272/0x2a0 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6447 ]--- >> Oct 24 13:02:01 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:01 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:01 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:01 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:01 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:01 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:01 host1 kernel: Call Trace: >> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:01 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:01 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:01 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:01 host1 kernel: [<ffffffff810a4cf7>] ? > futex_wait+0x227/0x380 >> Oct 24 13:02:01 host1 kernel: [<ffffffff81115a40>] ? >> generic_file_aio_read+0x0/0x700 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8117ad5b>] ? >> do_sync_readv_writev+0xfb/0x140 >> Oct 24 13:02:01 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:01 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8117be06>] ? >> do_readv_writev+0xd6/0x1f0 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 >> Oct 24 13:02:01 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:01 host1 kernel: ---[ end trace 4f331c7a492d6448 ]--- >> Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:02 host1 kernel: Call Trace: >> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8117afda>] ? > do_sync_read+0xfa/0x140 >> Oct 24 13:02:02 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 >> Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffff81083e26>] ? >> group_send_sig_info+0x56/0x70 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 >> Oct 24 13:02:02 host1 kernel: [<ffffffff810d6b12>] ? >> audit_syscall_entry+0x272/0x2a0 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d6449 ]--- >> Oct 24 13:02:02 host1 abrt-dump-oops: Reported 3 kernel oopses to Abrt >> Oct 24 13:02:02 host1 abrtd: Directory 'oops-2013-10-24-13:02:02-2922-2' >> creation detected >> Oct 24 13:02:02 host1 abrtd: Can't open file >> '/var/spool/abrt/oops-2013-10-24-13:02:02-2922-2/uid': No such file or >> directory >> Oct 24 13:02:02 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:02 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:02 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:02 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:02 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:02 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:02 host1 kernel: Call Trace: >> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:02 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:02 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:02 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:02 host1 kernel: [<ffffffff810a4cf7>] ? > futex_wait+0x227/0x380 >> Oct 24 13:02:02 host1 kernel: [<ffffffff81115a40>] ? >> generic_file_aio_read+0x0/0x700 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8117ad5b>] ? >> do_sync_readv_writev+0xfb/0x140 >> Oct 24 13:02:02 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:02 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8117be06>] ? >> do_readv_writev+0xd6/0x1f0 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 >> Oct 24 13:02:02 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:02 host1 kernel: ---[ end trace 4f331c7a492d644a ]--- >> Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:03 host1 kernel: Call Trace: >> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8117afda>] ? > do_sync_read+0xfa/0x140 >> Oct 24 13:02:03 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 >> Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffff81083e26>] ? >> group_send_sig_info+0x56/0x70 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 >> Oct 24 13:02:03 host1 kernel: [<ffffffff810d6b12>] ? >> audit_syscall_entry+0x272/0x2a0 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644b ]--- >> Oct 24 13:02:03 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:03 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:03 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:03 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:03 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:03 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:03 host1 kernel: Call Trace: >> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:03 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:03 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:03 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:03 host1 kernel: [<ffffffff810a4cf7>] ? > futex_wait+0x227/0x380 >> Oct 24 13:02:03 host1 kernel: [<ffffffff81115a40>] ? >> generic_file_aio_read+0x0/0x700 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8117ad5b>] ? >> do_sync_readv_writev+0xfb/0x140 >> Oct 24 13:02:03 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:03 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8117be06>] ? >> do_readv_writev+0xd6/0x1f0 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 >> Oct 24 13:02:03 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:03 host1 kernel: ---[ end trace 4f331c7a492d644c ]--- >> Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:04 host1 kernel: Call Trace: >> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8117afda>] ? > do_sync_read+0xfa/0x140 >> Oct 24 13:02:04 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 >> Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffff81083e26>] ? >> group_send_sig_info+0x56/0x70 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 >> Oct 24 13:02:04 host1 kernel: [<ffffffff810d6b12>] ? >> audit_syscall_entry+0x272/0x2a0 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644d ]--- >> Oct 24 13:02:04 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt >> Oct 24 13:02:04 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:04 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:04 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:04 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:04 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:04 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:04 host1 kernel: Call Trace: >> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:04 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:04 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:04 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:04 host1 kernel: [<ffffffff810a4cf7>] ? > futex_wait+0x227/0x380 >> Oct 24 13:02:04 host1 kernel: [<ffffffff81115a40>] ? >> generic_file_aio_read+0x0/0x700 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8117ad5b>] ? >> do_sync_readv_writev+0xfb/0x140 >> Oct 24 13:02:04 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:04 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8117be06>] ? >> do_readv_writev+0xd6/0x1f0 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 >> Oct 24 13:02:04 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:04 host1 kernel: ---[ end trace 4f331c7a492d644e ]--- >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 434865136 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 434865136 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 434865150 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 434865150 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 0 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 0 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 1 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 434865151 >> Oct 24 13:02:05 host1 kernel: Buffer I/O error on device dm-0, logical > block >> 1 >> Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:05 host1 kernel: Call Trace: >> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? > do_sync_read+0xfa/0x140 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 >> Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ? >> group_send_sig_info+0x56/0x70 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81083dd0>] ? >> group_send_sig_info+0x0/0x70 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ? >> audit_syscall_entry+0x272/0x2a0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d644f ]--- >> Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:05 host1 kernel: Pid: 28581, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:05 host1 kernel: Call Trace: >> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:05 host1 kernel: [<ffffffff811b13ed>] ? bio_endio+0x1d/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8112a965>] ? >> pagevec_lookup_tag+0x25/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:05 host1 kernel: ------------[ cut here ]------------ >> Oct 24 13:02:05 host1 kernel: WARNING: at fs/buffer.c:1199 >> __brelse+0x3a/0x40() (Tainted: G W --------------- ) >> Oct 24 13:02:05 host1 kernel: Hardware name: System x3630 M3 -[7377W03]- >> Oct 24 13:02:05 host1 kernel: VFS: brelse: Trying to free free buffer >> Oct 24 13:02:05 host1 kernel: Modules linked in: gfs2 dlm configfs ext3 > jbd >> nfs fscache ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE >> xt_state nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc >> cpufreq_ondemand acpi_cpufreq freq_table mperf bridge bonding 8021q garp > stp >> llc xt_CHECKSUM iptable_mangle ipt_REJECT iptable_filter iptable_nat > nf_nat >> nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables be2iscsi >> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio >> ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp >> libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net macvtap macvlan tun >> kvm_intel kvm ses enclosure sg e1000e igb cdc_ether usbnet mii microcode >> serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca >> i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi >> ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod >> [last unloaded: configfs] >> Oct 24 13:02:05 host1 kernel: Pid: 3864, comm: qemu-kvm Tainted: G > W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:05 host1 kernel: Call Trace: >> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b747>] ? >> warn_slowpath_common+0x87/0xc0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8106b836>] ? >> warn_slowpath_fmt+0x46/0x50 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81116a25>] ? > mempool_free+0x95/0xa0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff811ad48a>] ? __brelse+0x3a/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? > futex_wait+0x227/0x380 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa078bbc1>] ? >> gfs2_block_map+0x1e1/0xf30 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff81048fb5>] ? >> gup_pte_range+0xe5/0x130 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81115a40>] ? >> generic_file_aio_read+0x0/0x700 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117ad5b>] ? >> do_sync_readv_writev+0xfb/0x140 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81049148>] ? >> gup_pud_range+0x148/0x1d0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1770>] ? >> gfs2_get_block_direct+0x10/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811b689a>] ? >> __blockdev_direct_IO_newtrunc+0x85a/0xb90 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa0799ca8>] ? > do_promote+0x208/0x330 >> [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811b6c2e>] ? >> __blockdev_direct_IO+0x5e/0xd0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117be06>] ? >> do_readv_writev+0xd6/0x1f0 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1990>] ? >> gfs2_direct_IO+0x100/0x110 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117c133>] ? vfs_readv+0x43/0x60 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117c1f2>] ? sys_preadv+0xa2/0xc0 >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a1760>] ? >> gfs2_get_block_direct+0x0/0x20 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:05 host1 kernel: [<ffffffff810a4cf7>] ? > futex_wait+0x227/0x380 >> Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6450 ]--- >> Oct 24 13:02:05 host1 kernel: [<ffffffffa07a18ec>] ? >> gfs2_direct_IO+0x5c/0x110 [gfs2] >> Oct 24 13:02:05 host1 kernel: [<ffffffff811160fb>] ? >> generic_file_aio_read+0x6bb/0x700 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117afda>] ? > do_sync_read+0xfa/0x140 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81083a12>] ? send_signal+0x42/0x80 >> Oct 24 13:02:05 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81083e26>] ? >> group_send_sig_info+0x56/0x70 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8121fe8b>] ? >> selinux_file_permission+0xfb/0x150 >> Oct 24 13:02:05 host1 kernel: [<ffffffff81213236>] ? >> security_file_permission+0x16/0x20 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117b9e5>] ? vfs_read+0xb5/0x1a0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff810d6b12>] ? >> audit_syscall_entry+0x272/0x2a0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8117bd12>] ? sys_pread64+0x82/0xa0 >> Oct 24 13:02:05 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:05 host1 kernel: ---[ end trace 4f331c7a492d6451 ]--- >> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: fatal: I/O >> error >> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: block = >> 14244 >> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: function > = >> log_write_header, file = fs/gfs2/log.c, line = 616 >> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: about to >> withdraw this file system >> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: telling LM > to >> unmount >> Oct 24 13:02:06 host1 kernel: GFS2: fsid=ckvm1_pod1:vg_psv4.0: withdrawn >> Oct 24 13:02:06 host1 kernel: Pid: 4029, comm: parted Tainted: G W >> --------------- 2.6.32-279.5.1.el6.x86_64 #1 >> Oct 24 13:02:06 host1 kernel: Call Trace: >> Oct 24 13:02:06 host1 kernel: [<ffffffffa07b6062>] ? >> gfs2_lm_withdraw+0x102/0x130 [gfs2] >> Oct 24 13:02:06 host1 kernel: [<ffffffff814feb58>] ? >> out_of_line_wait_on_bit+0x78/0x90 >> Oct 24 13:02:06 host1 kernel: [<ffffffff81092110>] ? >> wake_bit_function+0x0/0x50 >> Oct 24 13:02:06 host1 kernel: [<ffffffffa07b60d0>] ? >> gfs2_io_error_bh_i+0x40/0x50 [gfs2] >> Oct 24 13:02:06 host1 kernel: [<ffffffff811ae0b6>] ? >> __wait_on_buffer+0x26/0x30 >> Oct 24 13:02:06 host1 kernel: [<ffffffffa079e288>] ? >> log_write_header+0x3a8/0x490 [gfs2] >> Oct 24 13:02:06 host1 kernel: [<ffffffffa079e951>] ? >> gfs2_log_flush+0x301/0x6f0 [gfs2] >> Oct 24 13:02:06 host1 kernel: [<ffffffff811a48a9>] ? >> sync_inodes_sb+0x149/0x190 >> Oct 24 13:02:06 host1 kernel: [<ffffffff810920d0>] ? >> autoremove_wake_function+0x0/0x40 >> Oct 24 13:02:06 host1 kernel: [<ffffffffa07b24cc>] ? > gfs2_sync_fs+0x2c/0x30 >> [gfs2] >> Oct 24 13:02:06 host1 kernel: [<ffffffff811aa2e3>] ? >> __sync_filesystem+0x53/0x90 >> Oct 24 13:02:06 host1 kernel: [<ffffffff811aa51b>] ? >> sync_filesystem+0x4b/0x70 >> Oct 24 13:02:06 host1 kernel: [<ffffffff811b514e>] ? fsync_bdev+0x2e/0x70 >> Oct 24 13:02:06 host1 kernel: [<ffffffff8125ea9b>] ? >> blkdev_ioctl+0x3ab/0x6e0 >> Oct 24 13:02:06 host1 kernel: [<ffffffff811b391c>] ? block_ioctl+0x3c/0x40 >> Oct 24 13:02:06 host1 kernel: [<ffffffff8118dff2>] ? vfs_ioctl+0x22/0xa0 >> Oct 24 13:02:06 host1 kernel: [<ffffffff8118e194>] ? > do_vfs_ioctl+0x84/0x580 >> Oct 24 13:02:06 host1 kernel: [<ffffffff8118e711>] ? sys_ioctl+0x81/0xa0 >> Oct 24 13:02:06 host1 kernel: [<ffffffff8100b0f2>] ? >> system_call_fastpath+0x16/0x1b >> Oct 24 13:02:06 host1 abrt-dump-oops: Reported 4 kernel oopses to Abrt >> Oct 24 13:02:29 host1 kernel: __ratelimit: 562 callbacks suppressed >> Oct 24 13:02:29 host1 kernel: kvm: 23115: cpu0 unhandled wrmsr: 0x198 data > 0 >> --- >8 --- >> >> Now I have this "uncontrolled entry" and it doesn't allow me to start cman >> again: >> --- 8< --- >> Oct 24 18:07:42 host1 dlm_controld[16712]: dlm_controld 3.0.12.1 started >> Oct 24 18:07:43 host1 gfs_controld[16761]: gfs_controld 3.0.12.1 started >> Oct 24 18:07:43 host1 gfs_controld[16761]: found uncontrolled gfs2 fs >> ckvm1_pod1:vg_psv4 >> Oct 24 18:07:43 host1 gfs_controld[16761]: telling cman to remove nodeid 1 >> from cluster >> Oct 24 18:07:43 host1 corosync[16628]: cman killed by node 1 because we > were >> killed by cman_tool or other application >> Oct 24 18:07:43 host1 dlm_controld[16712]: cluster is down, exiting >> Oct 24 18:07:43 host1 dlm_controld[16712]: daemon cpg_dispatch error 2 >> --- >8 --- >> >> How to get rid of it? How to make node to "forget" about this gfs2? >> >> Thank you! >> >> -----Original Message----- >> From: linux-cluster-bounces@xxxxxxxxxx >> [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Digimer >> Sent: Thursday, October 24, 2013 5:20 PM >> To: linux clustering >> Subject: Re: uncontrolled entry >> >> On 24/10/13 10:18, Vladimir Melnik wrote: >>> Dear colleagues, >>> >>> I'll be very grateful for any clue or advice. I have a cluster >>> (GFS2/DLM on iSCSI). Something happened today, so one node has stopped >>> working in cluster and all operations with the filesystem was giving >> "input/output errors". >>> Other nodes are working okay. >>> >>> I unmounted the filesystem and stopped all services (gfs2, clvmd, >>> cman, iscsi, iscsid), but I can't start cman again. That's what I have >>> in >>> log-files: >>> fenced[2801]: found uncontrolled entry >>> /sys/fs/gfs2/ckvm1_pod1:vg_psv4 >>> gfs_controld[2878]: found uncontrolled gfs2 fs ckvm1_pod1:vg_psv4 >>> >>> Yes, I really have this "uncontrolled entry", how to get rid of it >>> without rebooting the node? >>> >>> Thank you very much for any ideas and hints. >> >> I've not seen that particular error before. Can you paste your > cluster.conf >> and the log files from the effected node, starting just before the event >> occured? Please also look at the logs from other >> node(s) at the same time and paste how they responded to the node failure. >> >> -- >> Digimer >> Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is >> trapped in the mind of a person without access to education? >> >> -- >> Linux-cluster mailing list >> Linux-cluster@xxxxxxxxxx >> https://www.redhat.com/mailman/listinfo/linux-cluster >> > > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
