Mr Brown, Let me begin with: please give me your paypal address or something so that I can at least buy you a beer or something Your analysis and discovery that iSCSI is the origin of writes got me thinking: how can he see that on md0 device if that device has two more layers (bcache + LVM) before iSCSI even comes into play. Maybe the system propagates the origin down the block devices or something, totally not relevant here. So I embarked on a journey of total data destruction by disabling one layer at a time. I started by simply detaching bcache as that was the first thing on the list - and was non-destructive to boot :) I have found the culprit: It is bcache that does the one second writes. I have yet to find the exact parameters that influence this behaviour, but the output of writeback_rate_debug is EXTREMELY clear: it's writing a bit of data each second, reducing the dirty cache by that tiny amount. This is what causes the write "amplification" resulting in clicks long after a write has been done - because bcache only writes tiny amounts each second instead of flushing the entire cache at once when the time comes. Thank you for your time and please consider the first sentence of this mail LP, Jure On Tue, Dec 20, 2016 at 12:39 AM, NeilBrown <neilb@xxxxxxxx> wrote: > On Mon, Dec 19 2016, Jure Erznožnik wrote: > >> I have made two blktraces at the same time: one for md0 and one for a >> member of the array. I hope they will show something sensible. >> >> I dropped them here: >> http://expirebox.com/download/ee081fa4f85ffbd0bfad68e4ee257e11.html >> >> The file will be available for 48 hours or so they say. > > Thanks. > The blktrace_md0.txt file shows several writes that originate from > iscsi_trx > > so it looks like 'md0' (or some other device that is built on it) is > being exported as an iscsi target. Is that correct? > > There are also some large (1024 sector) writes being generated about > once per second. That is very clearly something being written to md0, > not something that md0 is doing itself. > They are generated by a kworker. It might be iscsi related, but it > might be something else. > > Try disabling the iscsi export and if the strange traffic still appears, > collect another blktrace (maybe for a bit longer - 30 seconds?). > Also please report output of > ls -l /sys/block/md0/holders/ > fuser /dev/md0 > > when the problem is occurring. > > NeilBrown -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
