On 00:38, Andre Noll wrote: > > I still don't think it's likely a filesystem problem but maybe you can > > pinpoint the fs behavior that triggers it. > > I'll try to reproduce the problem using different timeout values and the > ext4 options you suggest. If I can find a reliable reproducer, I'll run > blktrace and post the results. Here are some results. Prior to running the tests I wrote a bunch of 10G files and then filled the fs completely with 2T files containing zeros. Each of the tests below consisted of three runs of - remove 5 of the above 10G files to make 50G space available - run stress -d 5 --hdd-bytes 10G --hdd-noclean until it dies - run fsck if any fs errors occured Summary: Increasing the device timeout to 60s _or_ disabling barriers makes the problem go away. Deactivating delayed allocation makes the problem worse. - device timeout 60s, default ext4 parameters No problems at all, all three runs OK - device timeout 30s, default ext4 parameters 1. OK 2. dmesg: qla2xxx 0000:06:09.0: scsi(0:0:0): Abort command issued -- 1 2ea270b 2002. end_request: I/O error, dev sda, sector 7812889640 Aborting journal on device sda-8. EXT4-fs error (device sda): ext4_journal_start_sb: Detected aborted journal EXT4-fs (sda): Remounting filesystem read-only fsck: Inode 287, i_blocks is 4294918568, should be 416. Fix? yes Inode 288, i_size is 2198897426432, should be 2199023251456. Fix? yes Inode 288, i_blocks is 4294721960, should be 416. Fix? yes 3. qla2xxx 0000:06:09.0: scsi(0:0:0): Abort command issued -- 1 2ece6a8 2002. qla2xxx 0000:06:09.0: scsi(0:0:0): Abort command issued -- 1 2ece6dc 2002. end_request: I/O error, dev sda, sector 7812690136 Aborting journal on device sda-8. EXT4-fs error (device sda) in ext4_free_blocks: Journal has aborted EXT4-fs error (device sda) in ext4_ext_remove_space: Journal has aborted EXT4-fs error (device sda) in ext4_reserve_inode_write: Journal has aborted EXT4-fs error (device sda) in ext4_ext_truncate: Journal has aborted EXT4-fs error (device sda) in ext4_reserve_inode_write: Journal has aborted EXT4-fs error (device sda) in ext4_orphan_del: Journal has aborted EXT4-fs error (device sda) in ext4_reserve_inode_write: Journal has aborted EXT4-fs error (device sda) in ext4_delete_inode: Journal has aborted EXT4-fs error (device sda): ext4_journal_start_sb: Detected aborted journal EXT4-fs (sda): Remounting filesystem read-only fsck: e2fsck 1.41.10 (10-Feb-2009) /dev/sda: recovering journal Clearing orphaned inode 179 (uid=0, gid=0, mode=0100600, size=0) /dev/sda: clean, 301/244158464 files, 1935004235/1953247232 blocks - device timeout 30s, nodelalloc This seems to trigger the problem more reliably: 1. dmesg: same qla, ext4 errors as above fsck: orphaned inodes as above 2. and 3. errors already while removing files: rm: cannot remove `stress.98q1gG': Read-only file system dmesg: same qla/ext4 errors, but also JBD2: Detected IO errors while flushing file data on sda-8 fsck: clean - device timeout 30s, nobarrier No problem at all, all three runs OK. Eric, are you still interested in seeing the blktrace output? Suppose, I should use a 30s timeout, nodealloc and barriers=1 as this triggers the problem within minutes. Regards Andre -- The only person who always got his work done by Friday was Robinson Crusoe
Attachment:
signature.asc
Description: Digital signature
