I'm not certain this is LVM related but at this point I'm grasping at straws. Every so often, ranging from 14 days to a little over a month, the System will through the following error to messages: kernel: EXT3-fs error (device dm-6): ext3_readdir: directory #30933792 contains a hole at offset 28672 After that message shows up we get numerous "Journal Aborted" errors (messages clip at the bottom). The File system is marked Read-Only and that in turn initiates a fail-over of the application to the standby node. Then on the standby node, the filesystem is fsck'ed and is easily cleaned up without requiring any special intervention. Out SAN admin isn't seeing any errors on his end, and I don't see anything that jumps out at me in out messages. Every time this has happened the inode mentioned in the above error is different. We have rebuilt the VG and the subsequent filesystem from scratch once. we've turned the firmware parameters on the QLogic card to match the recommendations for EMC for our HW config and OS version. The failure occurs on both nodes (primary and standby) in the cluster. We've got a ticket open with EMC and another open with RedHat. Any help and/or ideas would be much appreciated.... Here's the system config: Red Hat Enterprise Linux AS release 4 (Nahant Update 2) Kernel: 2.6.9-22.ELsmp Qlogic HBA's Connected to an EMC Clariion Using QLogic's drivers and EMC Powerpath About 350Gig presented to the system LVM version: 2.01.14 (2005-08-04) Library version: 1.01.04 (2005-08-02) Driver version: 4.4.0 ------------------------------------------------------------------------------------------- # vgdisplay -v vg01 Using volume group(s) on command line Finding volume group "vg01" --- Volume group --- VG Name vg01 System ID xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Format lvm2 Metadata Areas 7 Metadata Sequence No 3 VG Access read/write VG Status resizable MAX LV 256 Cur LV 1 Open LV 1 Max PV 256 Cur PV 7 Act PV 7 VG Size 349.56 GB PE Size 32.00 MB Total PE 11186 Alloc PE / Size 11186 / 349.56 GB Free PE / Size 0 / 0 VG UUID 74biJE-Jn9V-i7jD-4Mng-2Pje-01YH-piE51u --- Logical volume --- LV Name /dev/vg01/data VG Name vg01 LV UUID 000000-0000-0000-0000-0000-0000-000000 LV Write Access read/write LV Status available # open 1 LV Size 349.56 GB Current LE 11186 Segments 7 Allocation normal Read ahead sectors 1024 Block device 253:6 --- Physical volumes --- PV Name /dev/emcpowera1 PV UUID DcRtrM-NtY6-ispA-uPwD-sJO1-SF0A-7iZlmm PV Status allocatable Total PE / Free PE 1598 / 0 PV Name /dev/emcpowerb1 PV UUID VIrhHw-LqEt-KNyZ-W86T-3oyY-9F25-Zu7zMA PV Status allocatable Total PE / Free PE 1598 / 0 PV Name /dev/emcpowerc1 PV UUID ySMNRg-m3XQ-ifF0-fWZx-ztAe-lhOX-seqJ9G PV Status allocatable Total PE / Free PE 1598 / 0 PV Name /dev/emcpowerd1 PV UUID puTa3Z-ZRrv-mRRS-orL0-Ndpz-dW4T-E1x46s PV Status allocatable Total PE / Free PE 1598 / 0 PV Name /dev/emcpowere1 PV UUID tsmFWq-YSVA-gHbI-ZGKG-4V57-766T-G8R6R2 PV Status allocatable Total PE / Free PE 1598 / 0 PV Name /dev/emcpowerf1 PV UUID WipA0D-GQXE-fDWL-OPNK-tRbx-7BUO-bAAJHY PV Status allocatable Total PE / Free PE 1598 / 0 PV Name /dev/emcpowerg1 PV UUID jbcB7h-qh8O-LkAB-wcJz-CxKy-GIIm-xwylk2 PV Status allocatable Total PE / Free PE 1598 / 0 ------------------------------------------------------------------------------------------- Messages Clip Below: ------------------------------------------------------------------------------------------- Dec 8 17:50:01 localhost logger: 17:50:01 up 17 days, 4:17, 2 users, load average: 3.07, 4.30, 4.54 Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6): ext3_readdir: directory #30933792 contains a hole at offset 28672 Dec 8 17:50:04 localhost kernel: Aborting journal on device dm-6. Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Readonly filesystem Dec 8 17:50:04 localhost kernel: Aborting journal on device dm-6. Dec 8 17:50:04 localhost kernel: ext3_abort called. Dec 8 17:50:04 localhost kernel: ext3_abort called. Dec 8 17:50:04 localhost kernel: ext3_abort called. Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6): ext3_journal_start_sb: Detected aborted journal Dec 8 17:50:04 localhost kernel: Remounting filesystem read-only Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6): ext3_journal_start_sb: <2>EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:04 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6): ext3_journal_start_sb: Detected aborted journalDetected aborted journal Dec 8 17:50:05 localhost kernel: Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost last message repeated 2 times Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_block: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_prepare_write: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost last message repeated 5 times Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_reserve_inode_write: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_inode: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_create: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost last message repeated 7 times Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_block: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_prepare_write: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost last message repeated 2 times Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_reserve_inode_write: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:05 localhost kernel: EXT3-fs error (device dm-6) in ext3_dirty_inode: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 11 times Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_block: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_prepare_write: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 7 times Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_reserve_inode_write: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_dirty_inode: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 5 times Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 6 times Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in add_dirent_to_buf: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_create: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_inode: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_create: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 37 times Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 44 times Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_inode: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_create: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 4 times Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_inode: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in ext3_create: Journal has aborted Dec 8 17:50:06 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:06 localhost last message repeated 3 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 8 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in ext3_ordered_commit_write: IO failure Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 3 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 16 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in add_dirent_to_buf: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in ext3_rename: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 6 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in ext3_new_block: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in ext3_prepare_write: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 7 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 3 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal hasEXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 3 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:07 localhost last message repeated 3 times Dec 8 17:50:07 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:08 localhost last message repeated 2 times Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:08 localhost last message repeated 2 times Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:08 localhost last message repeated 2 times Dec 8 17:50:08 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:08 localhost last message repeated 33 times Dec 8 17:50:08 localhost kernel: EXT3-fs freeing b_committed_data Dec 8 17:50:08 localhost kernel: __journal_remove_journal_head: freeing b_committed_data Dec 8 17:50:09 localhost last message repeated 26 times Dec 8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:09 localhost kernel: __journal_remove_journal_head: freeing b_committed_data Dec 8 17:50:09 localhost last message repeated 4 times Dec 8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:09 localhost kernel: __journal_remove_journal_head: freeing b_committed_data Dec 8 17:50:09 localhost last message repeated 34 times Dec 8 17:50:09 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:10 localhost last message repeated 1779 times Dec 8 17:50:11 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:25 localhost last message repeated 51 times Dec 8 17:50:25 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:26 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:27 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:36 localhost last message repeated 11 times Dec 8 17:50:38 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:50:56 localhost last message repeated 16 times Dec 8 17:51:01 localhost logger: 17:51:01 up 17 days, 4:18, 2 users, load average: 3.81, 4.62, 4.65 Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:51:04 localhost kernel: EXT3-fs error (device dm-6) in start_transaction: Journal has aborted Dec 8 17:51:17 localhost kernel: __journal_remove_journal_head: freeing b_committed_data Dec 8 17:51:17 localhost last message repeated 8 times Dec 8 17:52:01 localhost logger: 17:52:01 up 17 days, 4:19, 0 users, load average: 1.77, 3.92, 4.41 _______________________________________________ linux-lvm mailing list linux-lvm@redhat.com https://www.redhat.com/mailman/listinfo/linux-lvm read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
