Thanks for helping Neil. I have run the suggested utilities and here are my findings: It is always [kworker/x:yy] (x:yy changes somewhat) or [0]. A few lines from one of the outputs: 9,0 3 0 0.061577998 0 m N raid5 rcw 3758609392 2 2 0 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 9,0 0 1 0.065333879 283 C W 11275825480 [0] 9,0 0 1 0.065333879 283 C W 11275825480 [0] 9,0 0 1 0.065333879 283 C W 11275825480 [0] 9,0 0 1 0.065333879 283 C W 11275825480 [0] 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] 9,0 0 2 1.054590402 283 C W 11275826504 [0] 9,0 0 2 1.054590402 283 C W 11275826504 [0] 9,0 0 2 1.054590402 283 C W 11275826504 [0] 9,0 0 2 1.054590402 283 C W 11275826504 [0] 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 9,0 2 1 2.086924691 283 C W 11275861232 [0] 9,0 2 1 2.086924691 283 C W 11275861232 [0] 9,0 2 1 2.086924691 283 C W 11275861232 [0] 9,0 2 1 2.086924691 283 C W 11275861232 [0] 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 9,0 0 4 3.102340646 283 C W 11275861272 [0] 9,0 0 4 3.102340646 283 C W 11275861272 [0] 9,0 0 4 3.102340646 283 C W 11275861272 [0] 9,0 0 4 3.102340646 283 C W 11275861272 [0] 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 [kworker/3:38] 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 [kworker/3:38] 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 [kworker/3:38] 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 [kworker/3:38] 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 9,0 3 0 4.137872647 0 m N raid5 rcw 3758671448 2 0 1 9,0 3 0 4.137872647 0 m N raid5 rcw 3758671448 2 0 1 9,0 3 0 4.137872647 0 m N raid5 rcw 3758671448 2 0 1 LP, Jure On Wed, Dec 14, 2016 at 2:15 AM, NeilBrown <neilb@xxxxxxxx> wrote: > On Tue, Dec 13 2016, Jure Erznožnik wrote: > >> First of all, I apologise if this mail list is not intended for layman >> help, but this is what I am and I couldn't get an explanation >> elsewhere. >> >> My problem is that (as it seems) mdadm is touching HDD superblocks >> once per second, once at address 8 (sectors), next at address 16. >> Total traffic is kilobytes per second, writes only, no other >> detectable traffic. >> >> I have detailed the problem here: >> http://unix.stackexchange.com/questions/329477/ >> >> Shortened: >> kubuntu 16.10 4.8.0-30-generic #32, mdadm v3.4 2016-01-28 >> My configuration: 4 spinning platters (/dev/sd[cdef]) assembled into a >> raid5 array, then bcache set to cache (hopefully) everything >> (cache_mode = writeback, sequential_cutoff = 0). On top of bcache >> volume I have set up lvm. >> >> * iostat shows traffic on sd[cdef] and md0 >> * iotop shows no traffic >> * iosnoop shows COMM=[idle, md0_raid5, kworker] as processes working >> on the disk. Blocks reported are 8, 16 (data size a few KB) and >> 18446744073709500000 (data size 0). That last one must be some virtual >> thingie as the disks are nowhere near that large. >> * enabling block_dump shows md0_raid5 process writing to block 8 (1 >> sectors) and 16 (8 sectors) >> >> This touching is caused by any write into the array and goes on for >> quite a while after the write has been done (a couple of hours for >> 60GB of writes). When services actually work with the array, this >> becomes pretty much constant. >> >> What am I observing and is there any way of stopping it? > > Start with the uppermost layer which has I/O that you cannot explain. > Presumably that is md0. > Run 'blktrace' on that device for a little while, then 'blkparse' to > look at the results. > > blktrace -w 10 md0 > blkparse *blktrace* > > It will give the name of the process that initiated the request in [] at > the end of some lines. > > NeilBrown -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
