XFS Metadata corruption detected at xfs_attr3_leaf_write_verify

["Stockley, Jonathan" <jonathan.stockley@xxxxxxx>]

Hi,

I just ran into this error while testing an OpenStack SWIFT deployment.

[130004.933449] XFS (loop1): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x468d0c8

[130004.936209] XFS (loop1): Unmount and run xfs_repair

[130004.937477] XFS (loop1): First 64 bytes of corrupted metadata buffer:

[130004.939113] ffff880111ddd000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00  ................

[130004.941242] ffff880111ddd010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00  ..... ..........

[130004.943327] ffff880111ddd020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[130004.945393] ffff880111ddd030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[130004.947565] XFS (loop1): xfs_do_force_shutdown(0x8) called from line 1249 of file /build/linux-lts-vivid-vt3Z1H/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c.  Return address = 0xffffffffc0752c92

[130004.951692] XFS (loop1): Corruption of in-memory data detected.  Shutting down filesystem

Environment information:

Ubuntu Server 14.04 LTS

$ uname -a

Linux 3e2116e0-b4e8-4666-be70-5ddf9c9d9d2b 3.19.0-49-generic #55~14.04.1hf1533043v20160201b1-Ubuntu SMP Mon Feb 1 20:41:00 UT x86_64 x86_64 x86_64 GNU/Linux

I am able to reproduce the problem as follows:

• created a VM based SWIFT cluster
  One HAProxy load balancing across two SWIFT Proxy vms accessing five SWIFT storage nodes, although it could probably be simplified to one proxy and 1 storage node.
• Using ssbench with the following scenario file:
  {
    "name": "file upload only”,
    "sizes": [{
      "name": "files”,
      "size_min": 100000,
      "size_max": 100000
    }],
    "initial_files": {
      "files": 1
    },
    "container_count":10,
    "operation_count": 10000,
    "crud_profile": [50, 50, 0, 0],
    "user_count": 50
  }
• Run ssbench-master with following command line:
  ./ssbench-env/bin/ssbench-master run-scenario -f scenario1.json -A "http://aa.bb.cc.dd:8080/auth/v1.0" -U “acct:user" -K key --workers 10 --delete-after 36000 -r 18000

Replace aa.bb.cc.dd with either IP of HAProxy or SWIFT Proxy. Replace acct:user with SWIFT account and username. Replace key with user’s key (password). The test will run for 5 hours and objects will expire after 10 hours, but the test deletes all objects at the end of the run.

In my two test runs the XFS failure occurred around 9 hours after the test was started.

It looks like I can reproduce the problem, albeit over an extended period of time. 

What can I do to gather more info? Any debug options I can enable that might help?

Regards,

Jo Stockley.

_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs