Re: xfs blocks (blocked for more than 120 seconds)

["Michel Verbraak(1st-Setup)" <info@xxxxxxxxxxxx>]

op 24-03-14 18:36, Dave Chinner schreef:

On Mon, Mar 24, 2014 at 10:24:45AM +0100, Michel Verbraak(1st-Setup) wrote:

Hi,

We have a problem with one of our systems which is using XFS but we are
unable to find the problem. Recently we had two moments, Tuesday 4th of
March and Friday the 21st of March, where we had to reboot the system to
get it up and running again.

What happens:
- The programs handling files on the XFS disc stop working when
creating, deleting or writing files. They do not error they are just
waiting on the command to complete.
- One of our programs, a java application, goes into very high cpu usage
(50%) which normally is at 1%. This could be something in our java
application but it happens at the moment handling files gets stuck.
- A nice restart of the programs does not succeed as wel a kill -9 does
not work.
- Trying to reboot the servers in a normal fashion does not work. As it
is a virtual machine we have to do a shutdown (unplug power) and start
it up again to get it up and running.

......

Following details I have for you:

System OS: Ubuntu 12.04 LTS
Kernel: 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
Server: Virtual machine in a VMWare setup.
Disc: 300GB direct attached LUN

We have an exact clone of this system for our acceptance environment. In
this environment we are unable to reproduce this problem/situation.

Differences between the two days is that our services on 2014-03-21 were
quit busy with a lot of file changes on the xfs disc and on 2014-03-04
the system was very quiet on the moment the kernel traces appear and the
services get stuck.

Any help is appreciated.

Regards Michel Verbraak.

Following we see in the syslog on both moments (2014-03-04 and 2014-03-21):

....

Mar 21 06:32:20 ealxs00169 kernel: [1412280.930543] flush-8:16      D
0000000000000000     0 13864      2 0x00000000
[<ffffffff8165b34f>] schedule+0x3f/0x60
[<ffffffff8165b3ff>] io_schedule+0x8f/0xd0
[<ffffffff8111836e>] sleep_on_page+0xe/0x20
[<ffffffff8165baca>] __wait_on_bit_lock+0x5a/0xc0
[<ffffffff81118357>] __lock_page+0x67/0x70
[<ffffffff81122bd4>] write_cache_pages+0x3d4/0x460
[<ffffffff81122caa>] generic_writepages+0x4a/0x70
[<ffffffffa007980d>] xfs_vm_writepages+0x4d/0x60 [xfs]
[<ffffffff81123b71>] do_writepages+0x21/0x40
[<ffffffff811a2990>] writeback_single_inode+0x180/0x430
[<ffffffff811a3056>] writeback_sb_inodes+0x1b6/0x270
[<ffffffff811a31ae>] __writeback_inodes_wb+0x9e/0xd0
[<ffffffff811a345b>] wb_writeback+0x27b/0x330
[<ffffffff811a35af>] wb_check_old_data_flush+0x9f/0xb0
[<ffffffff811a4481>] wb_do_writeback+0x151/0x1d0
[<ffffffff811a4583>] bdi_writeback_thread+0x83/0x2a0
[<ffffffff8108b27c>] kthread+0x8c/0xa0

Writeback is blocked on a locked page, and is waiting for IO
completion.

We currently have the following options in fstab for the filesystem: defaults,noatime,inode64,barrier=no

Now I read everywhere to turn of barrier you should specify "nobarrier". Is our way of disabling wrong?

The disc used has "write cache" disabled:
[    2.875792] sd 3:0:0:0: Attached scsi generic sg2 type 0
[    2.876376] sd 3:0:0:0: [sdb] 629145600 512-byte logical blocks: (322 GB/300 GiB)
[    2.876879] sd 3:0:0:0: [sdb] Write Protect is off
[    2.877050] sd 3:0:0:0: [sdb] Mode Sense: 87 00 00 08
[    2.877890] sd 3:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    2.885634]  sdb: unknown partition table
...
[    5.132308] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[    5.155268] SGI XFS Quota Management subsystem
[    5.159345] XFS (sdb): Mounting Filesystem
..


Following is output of xfs_info on sdb:
meta-data="" isize=256    agcount=4, agsize=19660800 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=78643200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=38400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Mar 21 06:34:20 ealxs00169 kernel: [1412400.891181] archiver.pl     D
[<ffffffff8165b34f>] schedule+0x3f/0x60
[<ffffffff8165b995>] schedule_timeout+0x2a5/0x320
[<ffffffff8165c5f0>] __down_common+0xa5/0xf5
[<ffffffff8165c6b3>] __down+0x1d/0x1f

.....

Cheers,

Dave.

Michel.

_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs