Re: Tasks blocking forever with XFS stack traces

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

On Tue, 5 Nov 2019 at 08:54, Carlos Maiolino <cmaiolino@xxxxxxxxxx> wrote:
>
> Hi.
>
> On Tue, Nov 05, 2019 at 07:27:16AM +0000, Sitsofe Wheeler wrote:
> > Hi,
> >
> > We have a system that has been seeing tasks with XFS calls in their
> > stacks. Once these tasks start hanging with uninterruptible sleep any
> > write I/O to the directory they were doing I/O to will also hang
> > forever. The I/O they doing is being done to a bind mounted directory
> > atop an XFS filesystem on top an MD device (the MD device seems to be
> > still functional and isn't offline). The kernel is fairly old but I
> > thought I'd post a stack in case anyone can describe this or has seen
> > it before:
> >
> > kernel: [425684.110424] INFO: task kworker/u162:0:58843 blocked for
> > more than 120 seconds.
> > kernel: [425684.110800]       Tainted: G           OE
> > 4.15.0-64-generic #73-Ubuntu
> > kernel: [425684.111164] "echo 0 >
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > kernel: [425684.111568] kworker/u162:0  D    0 58843      2 0x80000080
> > kernel: [425684.111581] Workqueue: writeback wb_workfn (flush-9:126)
> > kernel: [425684.111585] Call Trace:
> > kernel: [425684.111595]  __schedule+0x24e/0x880
> > kernel: [425684.111664]  ? xfs_map_blocks+0x82/0x250 [xfs]

<snip>
> >
> > Other directories on the same filesystem seem fine as do other XFS
> > filesystems on the same system.
>
> The fact you mention other directories seems to work, and the first stack trace
> you posted, it sounds like you've been keeping a singe AG too busy to almost
> make it unusable. But, you didn't provide enough information we can really make
> any progress here, and to be honest I'm more inclined to point the finger to
> your MD device.

Let's see if we can pinpoint something :-)

> Can you describe your MD device? RAID array? What kind? How many disks?

RAID6 8 disks.

> What's your filesystem configuration? (xfs_info <mount point>)

meta-data=/dev/md126             isize=512    agcount=32, agsize=43954432 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0 rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=1406538240, imaxpct=5
         =                       sunit=128    swidth=768 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

> Do you have anything else on your dmesg other than these two stack traces? I'd
> suggest posting the whole dmesg, not only what you think is relevant.

Yes there's more. See a slightly elided dmesg from a longer run on
https://sucs.org/~sits/test/kern-20191024.log.gz .

>
> Better yet:
>
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Note most of the following was gathered from the currently not-hanging system:

kernel: was 4.15.0-64-generic from Ubuntu 18.04 but we're now testing
5.0.0-32-generic

xfsprogs version: xfs_repair version 4.9.0
CPUs: 80
cat /proc/meminfo
MemTotal:       791232512 kB
MemFree:        616987432 kB
MemAvailable:   781352708 kB
Buffers:            5520 kB
Cached:         113300540 kB
SwapCached:            0 kB
Active:         28385760 kB
Inactive:       85358040 kB
Active(anon):     436084 kB
Inactive(anon):     3476 kB
Active(file):   27949676 kB
Inactive(file): 85354564 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      31248380 kB
SwapFree:       31248380 kB
Dirty:               688 kB
Writeback:             0 kB
AnonPages:        436396 kB
Mapped:           206652 kB
Shmem:              6944 kB
KReclaimable:   56047960 kB
Slab:           58126044 kB
SReclaimable:   56047960 kB
SUnreclaim:      2078084 kB
KernelStack:       22240 kB
PageTables:        17552 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    426864636 kB
Committed_AS:    4147112 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:            61760 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     3245828 kB
DirectMap2M:    100208640 kB
DirectMap1G:    702545920 kB

cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs
rw,nosuid,relatime,size=395591264k,nr_inodes=98897816,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=79123252k,mode=755 0 0
/dev/mapper/vgsys-root / xfs rw,relatime,attr2,inode64,noquota 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/unified cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup
rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs
rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=66154
0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev 0 0
/dev/md0 /boot ext2 rw,relatime 0 0
/dev/md126 /localdata xfs
rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota 0 0
/dev/md126 /var/lib/docker xfs
rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota 0 0
/dev/mapper/vgsys-home /home xfs rw,relatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
overlay /var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XPFD5GLZ7YBMUP7S3E6W5OUE6A:/var/lib/docker/overlay2/l/GJVZ2MXOD5AOLUELAEYCSYCXLK:/var/lib/docker/overlay2/l/JEYWOT7MNNHX2DAE4AQ5XO674I:/var/lib/docker/overlay2/l/YAS2YWA4FTAWNEKRAJQY47TQDY,upperdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/diff,workdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/work,xino=off
0 0
overlay /localdata/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XPFD5GLZ7YBMUP7S3E6W5OUE6A:/var/lib/docker/overlay2/l/GJVZ2MXOD5AOLUELAEYCSYCXLK:/var/lib/docker/overlay2/l/JEYWOT7MNNHX2DAE4AQ5XO674I:/var/lib/docker/overlay2/l/YAS2YWA4FTAWNEKRAJQY47TQDY,upperdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/diff,workdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/work,xino=off
0 0
nsfs /run/docker/netns/160ed5c707bb nsfs rw 0 0
overlay /var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ECXX2YJFYUMBVKTP7OTRSAJVWE:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/diff,workdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/work,xino=off
0 0
overlay /localdata/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ECXX2YJFYUMBVKTP7OTRSAJVWE:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/diff,workdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/work,xino=off
0 0
nsfs /run/docker/netns/cc8ad7e2cc51 nsfs rw 0 0
overlay /var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/S5DDQ53MEAP37J6723CYPVDTO6:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/diff,workdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/work,xino=off
0 0
overlay /localdata/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/S5DDQ53MEAP37J6723CYPVDTO6:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/diff,workdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/work,xino=off
0 0
nsfs /run/docker/netns/e892b0d9fdea nsfs rw 0 0
overlay /var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/2MTUND5M3MS3FZWZCZVXTBIB5K:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/diff,workdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/work,xino=off
0 0
overlay /localdata/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/2MTUND5M3MS3FZWZCZVXTBIB5K:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/diff,workdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/work,xino=off
0 0
nsfs /run/docker/netns/e9d00dfcaa30 nsfs rw 0 0
overlay /var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SLHFVMXTCIQY5TYHXX3XY2QUTX:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/diff,workdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/work,xino=off
0 0
overlay /localdata/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SLHFVMXTCIQY5TYHXX3XY2QUTX:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/diff,workdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/work,xino=off
0 0
nsfs /run/docker/netns/2d3a60de14ae nsfs rw 0 0
tmpfs /run/user/2266 tmpfs
rw,nosuid,nodev,relatime,size=79123248k,mode=700,uid=2266,gid=501 0 0
tmpfs /run/user/2042 tmpfs
rw,nosuid,nodev,relatime,size=79123248k,mode=700,uid=2042,gid=501 0 0

cat /proc/partitions
major minor  #blocks  name

   8        0  937692504 sda
   8       16  937692504 sdb
   8       32  937692504 sdc
   8       48  937692504 sdd
   8       64  234431064 sde
   8       65     999424 sde1
   8       66          1 sde2
   8       69  233428992 sde5
   8       80  234431064 sdf
   8       81     999424 sdf1
   8       82          1 sdf2
   8       85  233428992 sdf5
   9      126 5626152960 md126
   9        0     998848 md0
   9        1  233297920 md1
   8       96  937692504 sdg
   8      112  937692504 sdh
   8      128  937692504 sdi
   8      144  937692504 sdj
 253        0  104857600 dm-0
 253        1   31248384 dm-1
 253        2   52428800 dm-2

cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath]
[raid0] [raid10]
md1 : active raid1 sdf5[1] sde5[0]
      233297920 blocks super 1.2 [2/2] [UU]
      bitmap: 1/2 pages [4KB], 65536KB chunk

md0 : active raid1 sdf1[1] sde1[0]
      998848 blocks super 1.2 [2/2] [UU]

md126 : active raid6 sdj[6] sdg[3] sdi[2] sdh[7] sdc[4] sdd[0] sda[5] sdb[1]
      5626152960 blocks level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
      bitmap: 0/7 pages [0KB], 65536KB chunk

unused devices: <none>

All disks are SATA Micron 5200 SSDs
No Battery Backed Write Cache

Workload:
Mixture of compiles and later on accelerator device I/O through
multiple docker containers. It usually takes days before the problem
is triggered.
I'm afraid I don't have the iostat/vmstat during the time of the
problem recorded

If there's key information missing that I can supply let me know and
I'll try and get it to you.

--
Sitsofe | http://sucs.org/~sits/




[Index of Archives]     [XFS Filesystem Development (older mail)]     [Linux Filesystem Development]     [Linux Audio Users]     [Yosemite Trails]     [Linux Kernel]     [Linux RAID]     [Linux SCSI]


  Powered by Linux