Hi,
On Tue, 5 Nov 2019 at 08:54, Carlos Maiolino <cmaiolino@xxxxxxxxxx> wrote:
>
> Hi.
>
> On Tue, Nov 05, 2019 at 07:27:16AM +0000, Sitsofe Wheeler wrote:
> > Hi,
> >
> > We have a system that has been seeing tasks with XFS calls in their
> > stacks. Once these tasks start hanging with uninterruptible sleep any
> > write I/O to the directory they were doing I/O to will also hang
> > forever. The I/O they doing is being done to a bind mounted directory
> > atop an XFS filesystem on top an MD device (the MD device seems to be
> > still functional and isn't offline). The kernel is fairly old but I
> > thought I'd post a stack in case anyone can describe this or has seen
> > it before:
> >
> > kernel: [425684.110424] INFO: task kworker/u162:0:58843 blocked for
> > more than 120 seconds.
> > kernel: [425684.110800] Tainted: G OE
> > 4.15.0-64-generic #73-Ubuntu
> > kernel: [425684.111164] "echo 0 >
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > kernel: [425684.111568] kworker/u162:0 D 0 58843 2 0x80000080
> > kernel: [425684.111581] Workqueue: writeback wb_workfn (flush-9:126)
> > kernel: [425684.111585] Call Trace:
> > kernel: [425684.111595] __schedule+0x24e/0x880
> > kernel: [425684.111664] ? xfs_map_blocks+0x82/0x250 [xfs]
<snip>
> >
> > Other directories on the same filesystem seem fine as do other XFS
> > filesystems on the same system.
>
> The fact you mention other directories seems to work, and the first stack trace
> you posted, it sounds like you've been keeping a singe AG too busy to almost
> make it unusable. But, you didn't provide enough information we can really make
> any progress here, and to be honest I'm more inclined to point the finger to
> your MD device.
Let's see if we can pinpoint something :-)
> Can you describe your MD device? RAID array? What kind? How many disks?
RAID6 8 disks.
> What's your filesystem configuration? (xfs_info <mount point>)
meta-data=/dev/md126 isize=512 agcount=32, agsize=43954432 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0 rmapbt=0
= reflink=0
data = bsize=4096 blocks=1406538240, imaxpct=5
= sunit=128 swidth=768 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=521728, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
> Do you have anything else on your dmesg other than these two stack traces? I'd
> suggest posting the whole dmesg, not only what you think is relevant.
Yes there's more. See a slightly elided dmesg from a longer run on
https://sucs.org/~sits/test/kern-20191024.log.gz .
>
> Better yet:
>
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
Note most of the following was gathered from the currently not-hanging system:
kernel: was 4.15.0-64-generic from Ubuntu 18.04 but we're now testing
5.0.0-32-generic
xfsprogs version: xfs_repair version 4.9.0
CPUs: 80
cat /proc/meminfo
MemTotal: 791232512 kB
MemFree: 616987432 kB
MemAvailable: 781352708 kB
Buffers: 5520 kB
Cached: 113300540 kB
SwapCached: 0 kB
Active: 28385760 kB
Inactive: 85358040 kB
Active(anon): 436084 kB
Inactive(anon): 3476 kB
Active(file): 27949676 kB
Inactive(file): 85354564 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 31248380 kB
SwapFree: 31248380 kB
Dirty: 688 kB
Writeback: 0 kB
AnonPages: 436396 kB
Mapped: 206652 kB
Shmem: 6944 kB
KReclaimable: 56047960 kB
Slab: 58126044 kB
SReclaimable: 56047960 kB
SUnreclaim: 2078084 kB
KernelStack: 22240 kB
PageTables: 17552 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 426864636 kB
Committed_AS: 4147112 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 61760 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 3245828 kB
DirectMap2M: 100208640 kB
DirectMap1G: 702545920 kB
cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs
rw,nosuid,relatime,size=395591264k,nr_inodes=98897816,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=79123252k,mode=755 0 0
/dev/mapper/vgsys-root / xfs rw,relatime,attr2,inode64,noquota 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/unified cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup
rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs
rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=66154
0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev 0 0
/dev/md0 /boot ext2 rw,relatime 0 0
/dev/md126 /localdata xfs
rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota 0 0
/dev/md126 /var/lib/docker xfs
rw,relatime,attr2,inode64,sunit=1024,swidth=6144,noquota 0 0
/dev/mapper/vgsys-home /home xfs rw,relatime,attr2,inode64,noquota 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
overlay /var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XPFD5GLZ7YBMUP7S3E6W5OUE6A:/var/lib/docker/overlay2/l/GJVZ2MXOD5AOLUELAEYCSYCXLK:/var/lib/docker/overlay2/l/JEYWOT7MNNHX2DAE4AQ5XO674I:/var/lib/docker/overlay2/l/YAS2YWA4FTAWNEKRAJQY47TQDY,upperdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/diff,workdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/work,xino=off
0 0
overlay /localdata/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XPFD5GLZ7YBMUP7S3E6W5OUE6A:/var/lib/docker/overlay2/l/GJVZ2MXOD5AOLUELAEYCSYCXLK:/var/lib/docker/overlay2/l/JEYWOT7MNNHX2DAE4AQ5XO674I:/var/lib/docker/overlay2/l/YAS2YWA4FTAWNEKRAJQY47TQDY,upperdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/diff,workdir=/var/lib/docker/overlay2/c86b0eab253a97ffe75b0661886337322c558386083bcb2d4823446025131b0a/work,xino=off
0 0
nsfs /run/docker/netns/160ed5c707bb nsfs rw 0 0
overlay /var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ECXX2YJFYUMBVKTP7OTRSAJVWE:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/diff,workdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/work,xino=off
0 0
overlay /localdata/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/ECXX2YJFYUMBVKTP7OTRSAJVWE:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/diff,workdir=/var/lib/docker/overlay2/551458a050177ebbc7b7e43646bc5cb645455cb6e9a5b1f420dc6b1a4322504d/work,xino=off
0 0
nsfs /run/docker/netns/cc8ad7e2cc51 nsfs rw 0 0
overlay /var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/S5DDQ53MEAP37J6723CYPVDTO6:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/diff,workdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/work,xino=off
0 0
overlay /localdata/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/S5DDQ53MEAP37J6723CYPVDTO6:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/diff,workdir=/var/lib/docker/overlay2/77096fc6ca39461683809377f6efa83957e73cdb91eb5f08957a64f75d829356/work,xino=off
0 0
nsfs /run/docker/netns/e892b0d9fdea nsfs rw 0 0
overlay /var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/2MTUND5M3MS3FZWZCZVXTBIB5K:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/diff,workdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/work,xino=off
0 0
overlay /localdata/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/2MTUND5M3MS3FZWZCZVXTBIB5K:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/diff,workdir=/var/lib/docker/overlay2/77b8012caabd1b32e965ba6258c4a41788a7e86e11205ec719d993f30a8e6257/work,xino=off
0 0
nsfs /run/docker/netns/e9d00dfcaa30 nsfs rw 0 0
overlay /var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SLHFVMXTCIQY5TYHXX3XY2QUTX:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/diff,workdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/work,xino=off
0 0
overlay /localdata/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/merged
overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SLHFVMXTCIQY5TYHXX3XY2QUTX:/var/lib/docker/overlay2/l/E4BBLB3NCC34KONYP23RP7VJ2X:/var/lib/docker/overlay2/l/SVYAOAODE6MEJVAEK2OO4SFF2E:/var/lib/docker/overlay2/l/A7TNW2Z7KHULNAU4BDB4GYRJ4A:/var/lib/docker/overlay2/l/SJ637O5BUZNAJSXNT27BO3CQGO:/var/lib/docker/overlay2/l/PYVRDDP7ABBFVD3PY2QGTJFQEM:/var/lib/docker/overlay2/l/OGFQOLFLSU27UIRKWXRZQ43OAP:/var/lib/docker/overlay2/l/KCOSL4MV3WQXKQZIZTQNTY4QEU:/var/lib/docker/overlay2/l/YTEXTILIATA6VFSWCQBWUHDY2D:/var/lib/docker/overlay2/l/4BAQ5SVXAVZWLTKZ6FH6VHJLWA:/var/lib/docker/overlay2/l/MUZSGTDT2THJSZEPFBG5NFWRGW:/var/lib/docker/overlay2/l/I6BCWJFX34IQ33OMCKNEHUUJU5:/var/lib/docker/overlay2/l/IRGYEAIEWEA4UJUYV3KEX3P4TI:/var/lib/docker/overlay2/l/J2PDWFCIYIFMH63PCXDJ6P2V7S:/var/lib/docker/overlay2/l/RC6FRWC3WRMRDRMCQM4L6R4VGA:/var/lib/docker/overlay2/l/HJM7E2PHDYPHGWF6RWP7R6OOZI:/var/lib/docker/overlay2/l/JI5RMXGTTBAM4NYEDR4FMNWV25:/var/lib/docker/overlay2/l/2TKWRPIAHOTDHLTGEYFRN4OUWL:/var/lib/docker/overlay2/l/6KCFDR62MDJOQ3ZA54IDNLUI7M:/var/lib/docker/overlay2/l/AN3SVYKAI6L4F54FKFSZMFDPUJ:/var/lib/docker/overlay2/l/YVJF7YEVLHXGC4L27UPEUK47HF:/var/lib/docker/overlay2/l/3NF7EYNTMPB7FFNI7POOBKXJPX:/var/lib/docker/overlay2/l/WAA6KYOATJLN6EP2PYYRQWEGOR:/var/lib/docker/overlay2/l/PHGIYF5LT5FKNUPFVSMEVHWNDU:/var/lib/docker/overlay2/l/KY5BSB7LSJPUNYBISCA4KYF7KS:/var/lib/docker/overlay2/l/HYDHRQJPMUKG4AXLIVBDPSUXJK:/var/lib/docker/overlay2/l/YI26DO7GTXPYQJSZ6BXHJUV5AR,upperdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/diff,workdir=/var/lib/docker/overlay2/28b0f26ad2c4dd1eccd966d1dc59499be968205a00572715db840abbbcc2789d/work,xino=off
0 0
nsfs /run/docker/netns/2d3a60de14ae nsfs rw 0 0
tmpfs /run/user/2266 tmpfs
rw,nosuid,nodev,relatime,size=79123248k,mode=700,uid=2266,gid=501 0 0
tmpfs /run/user/2042 tmpfs
rw,nosuid,nodev,relatime,size=79123248k,mode=700,uid=2042,gid=501 0 0
cat /proc/partitions
major minor #blocks name
8 0 937692504 sda
8 16 937692504 sdb
8 32 937692504 sdc
8 48 937692504 sdd
8 64 234431064 sde
8 65 999424 sde1
8 66 1 sde2
8 69 233428992 sde5
8 80 234431064 sdf
8 81 999424 sdf1
8 82 1 sdf2
8 85 233428992 sdf5
9 126 5626152960 md126
9 0 998848 md0
9 1 233297920 md1
8 96 937692504 sdg
8 112 937692504 sdh
8 128 937692504 sdi
8 144 937692504 sdj
253 0 104857600 dm-0
253 1 31248384 dm-1
253 2 52428800 dm-2
cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4] [linear] [multipath]
[raid0] [raid10]
md1 : active raid1 sdf5[1] sde5[0]
233297920 blocks super 1.2 [2/2] [UU]
bitmap: 1/2 pages [4KB], 65536KB chunk
md0 : active raid1 sdf1[1] sde1[0]
998848 blocks super 1.2 [2/2] [UU]
md126 : active raid6 sdj[6] sdg[3] sdi[2] sdh[7] sdc[4] sdd[0] sda[5] sdb[1]
5626152960 blocks level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
bitmap: 0/7 pages [0KB], 65536KB chunk
unused devices: <none>
All disks are SATA Micron 5200 SSDs
No Battery Backed Write Cache
Workload:
Mixture of compiles and later on accelerator device I/O through
multiple docker containers. It usually takes days before the problem
is triggered.
I'm afraid I don't have the iostat/vmstat during the time of the
problem recorded
If there's key information missing that I can supply let me know and
I'll try and get it to you.
--
Sitsofe | http://sucs.org/~sits/
