Hello All, Kind reminder that I had to start a similar thread last month. https://marc.info/?t=147871214200005&r=1&w=2 Just in case it rings any bells. BR Theo --- Best regards, ΜΦΧ, Theophanis Kontogiannis On Sun, Dec 18, 2016 at 9:40 PM, Jure Erznožnik <jure.erznoznik@xxxxxxxxx> wrote: > My further attempts to solve this issue include the following (all > unsuccessful): > > 1. Installing a fresh Ubuntu, assemble the array > 2. Install OpenSUSE, assemble the array > 3. Tear the array down, create it anew from scratch (it now has a new > UUID, but the data seems to have been preserved, so my bcache / LVM2 > configuration remains the same) - interestingly though: during initial > array rebuild which took the better part of today, there was no > clicking even though drives were constantly in action. Either it was > inaudible or the touching didn't take place. > > I think I'm barking up the wrong tree with these experiments. Not sure > how to proceed from here. > > LP, > Jure > > On Thu, Dec 15, 2016 at 8:01 AM, Jure Erznožnik > <jure.erznoznik@xxxxxxxxx> wrote: >> Thanks for helping Neil. I have run the suggested utilities and here >> are my findings: >> >> It is always [kworker/x:yy] (x:yy changes somewhat) or [0]. >> A few lines from one of the outputs: >> >> 9,0 3 0 0.061577998 0 m N raid5 rcw 3758609392 2 2 0 >> 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 >> 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 >> 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 >> 9,0 3 0 0.061580084 0 m N raid5 rcw 3758609400 2 2 0 >> 9,0 0 1 0.065333879 283 C W 11275825480 [0] >> 9,0 0 1 0.065333879 283 C W 11275825480 [0] >> 9,0 0 1 0.065333879 283 C W 11275825480 [0] >> 9,0 0 1 0.065333879 283 C W 11275825480 [0] >> 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] >> 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] >> 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] >> 9,0 3 2 1.022155200 2861 Q W 11275826504 + 32 [kworker/3:38] >> 9,0 0 2 1.054590402 283 C W 11275826504 [0] >> 9,0 0 2 1.054590402 283 C W 11275826504 [0] >> 9,0 0 2 1.054590402 283 C W 11275826504 [0] >> 9,0 0 2 1.054590402 283 C W 11275826504 [0] >> 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] >> 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] >> 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] >> 9,0 3 3 2.046065106 2861 Q W 11275861232 + 8 [kworker/3:38] >> 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 >> 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 >> 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 >> 9,0 0 0 2.075247515 0 m N raid5 rcw 3758619888 2 0 1 >> 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 >> 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 >> 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 >> 9,0 0 0 2.075250686 0 m N raid5 rcw 3758619888 2 2 0 >> 9,0 2 1 2.086924691 283 C W 11275861232 [0] >> 9,0 2 1 2.086924691 283 C W 11275861232 [0] >> 9,0 2 1 2.086924691 283 C W 11275861232 [0] >> 9,0 2 1 2.086924691 283 C W 11275861232 [0] >> 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] >> 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] >> 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] >> 9,0 0 3 2.967340614 1061 Q FWS [kworker/0:18] >> 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] >> 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] >> 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] >> 9,0 3 4 3.070092310 2861 Q W 11275861272 + 8 [kworker/3:38] >> 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 >> 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 >> 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 >> 9,0 0 0 3.101966398 0 m N raid5 rcw 3758619928 2 0 1 >> 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 >> 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 >> 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 >> 9,0 0 0 3.101969169 0 m N raid5 rcw 3758619928 2 2 0 >> 9,0 0 4 3.102340646 283 C W 11275861272 [0] >> 9,0 0 4 3.102340646 283 C W 11275861272 [0] >> 9,0 0 4 3.102340646 283 C W 11275861272 [0] >> 9,0 0 4 3.102340646 283 C W 11275861272 [0] >> 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 >> [kworker/3:38] >> 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 >> [kworker/3:38] >> 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 >> [kworker/3:38] >> 9,0 3 5 4.094666938 2861 Q W 11276014160 + 336 >> [kworker/3:38] >> 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 >> 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 >> 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 >> 9,0 3 0 4.137869804 0 m N raid5 rcw 3758671440 2 0 1 >> 9,0 3 0 4.137872647 0 m N raid5 rcw 3758671448 2 0 1 >> 9,0 3 0 4.137872647 0 m N raid5 rcw 3758671448 2 0 1 >> 9,0 3 0 4.137872647 0 m N raid5 rcw 3758671448 2 0 1 >> >> LP, >> Jure >> >> On Wed, Dec 14, 2016 at 2:15 AM, NeilBrown <neilb@xxxxxxxx> wrote: >>> On Tue, Dec 13 2016, Jure Erznožnik wrote: >>> >>>> First of all, I apologise if this mail list is not intended for layman >>>> help, but this is what I am and I couldn't get an explanation >>>> elsewhere. >>>> >>>> My problem is that (as it seems) mdadm is touching HDD superblocks >>>> once per second, once at address 8 (sectors), next at address 16. >>>> Total traffic is kilobytes per second, writes only, no other >>>> detectable traffic. >>>> >>>> I have detailed the problem here: >>>> http://unix.stackexchange.com/questions/329477/ >>>> >>>> Shortened: >>>> kubuntu 16.10 4.8.0-30-generic #32, mdadm v3.4 2016-01-28 >>>> My configuration: 4 spinning platters (/dev/sd[cdef]) assembled into a >>>> raid5 array, then bcache set to cache (hopefully) everything >>>> (cache_mode = writeback, sequential_cutoff = 0). On top of bcache >>>> volume I have set up lvm. >>>> >>>> * iostat shows traffic on sd[cdef] and md0 >>>> * iotop shows no traffic >>>> * iosnoop shows COMM=[idle, md0_raid5, kworker] as processes working >>>> on the disk. Blocks reported are 8, 16 (data size a few KB) and >>>> 18446744073709500000 (data size 0). That last one must be some virtual >>>> thingie as the disks are nowhere near that large. >>>> * enabling block_dump shows md0_raid5 process writing to block 8 (1 >>>> sectors) and 16 (8 sectors) >>>> >>>> This touching is caused by any write into the array and goes on for >>>> quite a while after the write has been done (a couple of hours for >>>> 60GB of writes). When services actually work with the array, this >>>> becomes pretty much constant. >>>> >>>> What am I observing and is there any way of stopping it? >>> >>> Start with the uppermost layer which has I/O that you cannot explain. >>> Presumably that is md0. >>> Run 'blktrace' on that device for a little while, then 'blkparse' to >>> look at the results. >>> >>> blktrace -w 10 md0 >>> blkparse *blktrace* >>> >>> It will give the name of the process that initiated the request in [] at >>> the end of some lines. >>> >>> NeilBrown > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
