|
We've run into a bit of an issue with xfs running Ceph. The following
bug details what we are seeing:
https://bugs.launchpad.net/ubuntu/+source/xfs/+bug/1464308 Basically the Ceph OSD process gets hung in dstate due to the traceback in the bug. Here is additional info gathered: xfs bmap output for a random directory https://gist.github.com/dmmatson/e864252c7ff346df954a attr -l of the file dchinner indicated from the xfs bmap output (attr -l) (6:11:41 PM) dmatson: Attribute "cephos.spill_out" has a 2 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5 (6:11:45 PM) dmatson: Attribute "ceph.snapset@3" has a 263 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5 (6:11:49 PM) dmatson: Attribute "ceph.snapset@2" has a 1131 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5 (6:11:53 PM) dmatson: Attribute "ceph.snapset@1" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5 (6:11:56 PM) dmatson: Attribute "ceph._" has a 259 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5 (6:12:00 PM) dmatson: Attribute "ceph.snapset" has a 2048 byte value for rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5 xfs_bmap -vp of same file rbd\udata.66039648e29a80.0000000000000d35__head_23EA10B8__5: (6:13:21 PM) dmatson: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS (6:13:25 PM) dmatson: 0: [0..8191]: 2944471376..2944479567 16 (24445776..24453967) 8192 00000 uname -a Linux str-slc-02-01 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux xfsprogs version xfs_repair version 3.1.9 number of cpus 2x6 core /proc/meminfo MemTotal: 264123244 kB MemFree: 115763776 kB MemAvailable: 118825516 kB Buffers: 8624 kB Cached: 5331460 kB SwapCached: 0 kB Active: 137378292 kB Inactive: 4797536 kB Active(anon): 136839564 kB Inactive(anon): 5160 kB Active(file): 538728 kB Inactive(file): 4792376 kB Unevictable: 8576 kB Mlocked: 8896 kB SwapTotal: 34058232 kB SwapFree: 34058232 kB Dirty: 116400 kB Writeback: 3360 kB AnonPages: 136847296 kB Mapped: 3003196 kB Shmem: 2776 kB Slab: 1348116 kB SReclaimable: 581188 kB SUnreclaim: 766928 kB KernelStack: 1788400 kB PageTables: 505196 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 166119852 kB Committed_AS: 259735308 kB VmallocTotal: 34359738367 kB VmallocUsed: 908732 kB VmallocChunk: 34221991936 kB HardwareCorrupted: 0 kB AnonHugePages: 34844672 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 137500 kB DirectMap2M: 5070848 kB DirectMap1G: 265289728 kB rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,relatime,size=132050356k,nr_inodes=33012589,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=26412328k,mode=755 0 0 /dev/mapper/system-root / ext4 rw,relatime,discard,errors=remount-ro,data="" 0 0 none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0 none /sys/fs/fuse/connections fusectl rw,relatime 0 0 none /sys/kernel/debug debugfs rw,relatime 0 0 none /sys/kernel/security securityfs rw,relatime 0 0 none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0 none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0 none /sys/fs/pstore pstore rw,relatime 0 0 /dev/sda1 /boot ext3 rw,relatime,data="" 0 0 systemd /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,name=systemd 0 0 /dev/mapper/e0364357-e2e0-4b5e-92c3-abbea655ca77 /var/lib/ceph/osd/ceph-293 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/a7745ae8-1b83-4cb5-892b-4cc8d27ed5b2 /var/lib/ceph/osd/ceph-302 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/60cecd8c-42fd-4cbf-9264-ead7319846d3 /var/lib/ceph/osd/ceph-295 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/c3ab6273-a646-40fe-bbf3-c4559bf4202a /var/lib/ceph/osd/ceph-298 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/3dfceed9-6563-46c9-b174-7173f73874ae /var/lib/ceph/osd/ceph-301 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/5765a366-3908-41ef-b2ab-dd10db6e8c29 /var/lib/ceph/osd/ceph-299 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/bb5b11a9-cc4d-49b8-a909-c8e84a88f0ee /var/lib/ceph/osd/ceph-316 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/4d3165f9-d879-426f-b7ef-6a959a09a1bf /var/lib/ceph/osd/ceph-313 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/tmp/mnt.BeT4xM xfs rw,relatime,attr2,inode64,noquota 0 0 /dev/mapper/b492dfe2-103c-44b6-83ad-8dc35a9818c0 /var/lib/ceph/osd/ceph-303 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/e5b359e2-bb98-40a5-a285-612b8fb2b2a6 /var/lib/ceph/osd/ceph-314 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/6b45df8f-b6c0-4ed7-b791-a2694b3212ec /var/lib/ceph/osd/ceph-312 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/abcd86ff-57d0-4ef0-8ee7-1a13408b3b2c /var/lib/ceph/osd/ceph-322 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/a75ed2c0-cc39-4063-bb1e-202d47798499 /var/lib/ceph/osd/ceph-317 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/4d788f02-2720-4a50-8bac-e5a4cdf99441 /var/lib/ceph/osd/ceph-321 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/8d179b94-a071-4dab-927b-b9d2ed98451d /var/lib/ceph/osd/ceph-308 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/b781ebc5-7259-4cf2-8fd5-c608c61fe0e2 /var/lib/ceph/osd/ceph-318 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/deaa099e-9791-4c03-826e-6386eb2c4fff /var/lib/ceph/osd/ceph-323 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/bcbf31e3-7f17-49aa-9259-6dfe8e0f04bc /var/lib/ceph/osd/ceph-320 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/d2b9331f-bcfa-4c98-a0b7-5ab1ec93ee02 /var/lib/ceph/osd/ceph-311 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/0c4172f9-9acb-4c88-bc4e-1b6eb898ea28 /var/lib/ceph/osd/ceph-325 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/252ee856-0376-44ae-b6d7-03f930f8d082 /var/lib/ceph/osd/ceph-328 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/d4bcc76c-20c3-4253-bd05-472f3eb7df96 /var/lib/ceph/osd/ceph-319 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/2ca3f1d3-cf59-49df-a008-0bb6c549587b /var/lib/ceph/osd/ceph-315 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/16df1fdc-819b-4593-ac4f-80f3fcac6054 /var/lib/ceph/osd/ceph-306 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/ce83d72e-029b-40b6-9d59-3b0ebadea2c6 /var/lib/ceph/osd/ceph-324 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/86e078fe-6fa7-466c-83eb-14c53b531320 /var/lib/ceph/osd/ceph-307 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/368ef2f0-da43-48ee-bd91-2bfba95aea5a /var/lib/ceph/osd/ceph-310 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/ac67fd93-8145-40ce-96bb-812cec326306 /var/lib/ceph/osd/ceph-327 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/098e2832-e9d4-4b84-9fa9-acfa407a3afc /var/lib/ceph/osd/ceph-309 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/f1588472-430d-4db4-83dc-31e1ec1d2ba5 /var/lib/ceph/osd/ceph-326 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/fb64b259-7d70-4859-824e-b6ee7683495d /var/lib/ceph/osd/ceph-305 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/2559e439-9540-40eb-99a6-cebbe2663cd9 /var/lib/ceph/osd/ceph-300 xfs rw,noatime,attr2,inode64,noquota 0 0 /dev/mapper/ae8b44b1-25ac-4aa9-b987-9c57c3410ec6 /var/lib/ceph/osd/ceph-304 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/c198f472-4806-4c9c-aa79-b8e1b5554cdf /var/lib/ceph/osd/ceph-296 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/65d17dba-4c30-4e95-a0b5-577002712b6a /var/lib/ceph/osd/ceph-297 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 /dev/mapper/1f3dc9e8-0d84-4951-8c9b-48b6a6be83b8 /var/lib/ceph/osd/ceph-294 xfs rw,noatime,attr2,inode64,allocsize=4096k,noquota 0 0 major minor #blocks name 8 0 125034840 sda 8 1 291840 sda1 8 2 1 sda2 8 5 124740608 sda5 252 0 124235776 dm-0 252 1 503808 dm-1 8 32 2930266584 sdc 8 33 2920025543 sdc1 8 34 10238976 sdc2 8 128 2930266584 sdi 8 129 2920025543 sdi1 8 130 10238976 sdi2 8 96 2930266584 sdg 8 97 2920025543 sdg1 8 98 10238976 sdg2 8 112 2930266584 sdh 8 113 2920025543 sdh1 8 114 10238976 sdh2 8 176 2930266584 sdl 8 177 2920025543 sdl1 8 178 10238976 sdl2 8 144 2930266584 sdj 8 145 2920025543 sdj1 8 146 10238976 sdj2 8 240 2930266584 sdp 8 241 2920025543 sdp1 8 242 10238976 sdp2 8 224 2930266584 sdo 8 225 2920025543 sdo1 8 226 10238976 sdo2 8 80 2930266584 sdf 8 81 2920025543 sdf1 8 82 10238976 sdf2 8 48 2930266584 sdd 8 49 2920025543 sdd1 8 50 10238976 sdd2 65 48 2930266584 sdt 65 49 2920025543 sdt1 65 50 10238976 sdt2 8 64 2930266584 sde 8 65 2920025543 sde1 8 66 10238976 sde2 65 0 2930266584 sdq 65 1 2920025543 sdq1 65 2 10238976 sdq2 8 192 2930266584 sdm 8 193 2920025543 sdm1 8 194 10238976 sdm2 65 32 2930266584 sds 65 33 2920025543 sds1 65 34 10238976 sds2 65 64 2930266584 sdu 65 65 2920025543 sdu1 65 66 10238976 sdu2 65 16 2930266584 sdr 65 17 2920025543 sdr1 65 18 10238976 sdr2 65 160 2930266584 sdaa 65 161 2920025543 sdaa1 65 162 10238976 sdaa2 8 160 2930266584 sdk 8 161 2920025543 sdk1 8 162 10238976 sdk2 65 240 2930266584 sdaf 65 241 2920025543 sdaf1 65 242 10238976 sdaf2 8 16 2930266584 sdb 8 17 2920025543 sdb1 8 18 10238976 sdb2 65 176 2930266584 sdab 65 177 2920025543 sdab1 65 178 10238976 sdab2 65 96 2930266584 sdw 65 97 2920025543 sdw1 65 98 10238976 sdw2 65 128 2930266584 sdy 65 129 2920025543 sdy1 65 130 10238976 sdy2 66 16 2930266584 sdah 66 17 2920025543 sdah1 66 18 10238976 sdah2 8 208 2930266584 sdn 8 209 2920025543 sdn1 8 210 10238976 sdn2 65 144 2930266584 sdz 65 145 2920025543 sdz1 65 146 10238976 sdz2 65 80 2930266584 sdv 65 81 2920025543 sdv1 65 82 10238976 sdv2 65 192 2930266584 sdac 65 193 2920025543 sdac1 65 194 10238976 sdac2 65 224 2930266584 sdae 65 225 2920025543 sdae1 65 226 10238976 sdae2 65 112 2930266584 sdx 65 113 2920025543 sdx1 65 114 10238976 sdx2 66 0 2930266584 sdag 66 1 2920025543 sdag1 66 2 10238976 sdag2 65 208 2930266584 sdad 65 209 2920025543 sdad1 65 210 10238976 sdad2 66 64 2930266584 sdak 66 65 2920025543 sdak1 66 66 10238976 sdak2 66 48 2930266584 sdaj 66 49 2920025543 sdaj1 66 50 10238976 sdaj2 66 32 2930266584 sdai 66 33 2920025543 sdai1 66 34 10238976 sdai2 252 2 10238976 dm-2 252 3 2920025543 dm-3 252 4 10238976 dm-4 252 5 2920025543 dm-5 252 6 2920025543 dm-6 252 7 10238976 dm-7 252 8 2920025543 dm-8 252 9 10238976 dm-9 252 10 2920025543 dm-10 252 11 10238976 dm-11 252 12 10238976 dm-12 252 13 2920025543 dm-13 252 14 10238976 dm-14 252 15 2920025543 dm-15 252 16 10238976 dm-16 252 17 2920025543 dm-17 252 18 2920025543 dm-18 252 19 10238976 dm-19 252 20 2920025543 dm-20 252 21 10238976 dm-21 252 22 10238976 dm-22 252 23 2920025543 dm-23 252 24 10238976 dm-24 252 25 2920025543 dm-25 252 26 2920025543 dm-26 252 27 10238976 dm-27 252 28 10238976 dm-28 252 29 2920025543 dm-29 252 31 2920025543 dm-31 252 30 10238976 dm-30 252 32 10238976 dm-32 252 33 2920025543 dm-33 252 34 10238976 dm-34 252 35 2920025543 dm-35 252 36 2920025543 dm-36 252 37 10238976 dm-37 252 38 2920025543 dm-38 252 39 10238976 dm-39 252 40 2920025543 dm-40 252 41 2920025543 dm-41 252 42 2920025543 dm-42 252 43 10238976 dm-43 252 44 10238976 dm-44 252 45 10238976 dm-45 252 46 10238976 dm-46 252 47 2920025543 dm-47 252 48 2920025543 dm-48 252 49 10238976 dm-49 252 50 2920025543 dm-50 252 51 10238976 dm-51 252 52 2920025543 dm-52 252 53 10238976 dm-53 252 54 10238976 dm-54 252 55 2920025543 dm-55 252 56 2920025543 dm-56 252 57 10238976 dm-57 252 59 2920025543 dm-59 252 58 10238976 dm-58 252 60 10238976 dm-60 252 61 2920025543 dm-61 252 62 2920025543 dm-62 252 63 10238976 dm-63 252 64 2920025543 dm-64 252 65 10238976 dm-65 252 66 2920025543 dm-66 252 67 10238976 dm-67 252 68 2920025543 dm-68 252 69 10238976 dm-69 252 70 10238976 dm-70 252 71 2920025543 dm-71 252 72 10238976 dm-72 252 73 2920025543 dm-73 RAID layout: None LVM configuration: None HGST 3TB meta-data="" isize=2048 agcount=32, agsize=22812700 blks = sectsz=4096 attr=2 data = bsize=4096 blocks=730006385, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal bsize=4096 blocks=356448, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Some additional info can be found in the following ubuntu bug:
If you are not the intended recipient of this message, be advised that any dissemination or copying of this message is prohibited.
If you received this message erroneously, please notify the sender and delete it, together with any attachments. |
Attachment:
dmesg.out
Description: dmesg.out
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
