humm, let´s wait anothers ideas from list 2011/3/18 Hari Subramanian <hari@xxxxxxxxxx>: > Roberto, I still think the solution you point out has the potential for throttling foreground IOs issued to MD from the filesystem as well as the MD initiated background resyncs. So, I don't want to limit the IO queues, esp since our foreground workload involves a LOT of small random IO. > > Thanks > ~ Hari > > -----Original Message----- > From: rspadim@xxxxxxxxx [mailto:rspadim@xxxxxxxxx] On Behalf Of Roberto Spadim > Sent: Friday, March 18, 2011 4:36 PM > To: Hari Subramanian > Cc: linux-raid@xxxxxxxxxxxxxxx > Subject: Re: Adaptive throttling for RAID1 background resync > > hum, it´s not a io queue size (very big ram memory queue) problem? > maybe getting it smaller could help? > resync is something like read here write there, if you have write > problem, read should stop when async writes can´t work more (no ram > memory) > i´m right? if true, that´s why i think queue is a point to check > > 2011/3/18 Hari Subramanian <hari@xxxxxxxxxx>: >> Roberto, My use case involves both foreground and background resyncs happening at the same time. So, by throttling it at the block or IO queues, I would be limiting my throughout for foreground IOs as well which is undesirable. >> >> ~ Hari >> >> -----Original Message----- >> From: rspadim@xxxxxxxxx [mailto:rspadim@xxxxxxxxx] On Behalf Of Roberto Spadim >> Sent: Friday, March 18, 2011 4:29 PM >> To: Hari Subramanian >> Cc: linux-raid@xxxxxxxxxxxxxxx >> Subject: Re: Adaptive throttling for RAID1 background resync >> >> maybe this could be better solved at queue linux kernel area... at >> elevators or block devices >> >> 2011/3/18 Hari Subramanian <hari@xxxxxxxxxx>: >>> I am hitting an issue when performing RAID1 resync from a replica hosted on a fast disk to one on a slow disk. When resync throughput is set at 20Mbps min and 200Mbps max and we have enough data to resync, I see the kernel running out of memory quickly (within a minute). From the crash dumps, I see that a whole lot (12,000+) of biovec-64s that are active on the slab cache. >>> >>> Our guess is that MD is allowing data to be read from the fast disk at a frequency much higher than what the slow disk is able to write to. This continues for a long time (> 1 minute) in an unbounded fashion resulting in buildup of IOs that are waiting to be written to the disk. This eventually causes the machine to panic (we have panic on OOM selected) >>> >>> From reading the MD and RAID1 resync code, I don't see anything that would prevent something like this from happening. So, we would like to implement something to this effect that adaptively throttles the background resync. >>> >>> Can someone confirm or deny these claims and also the need for a new solution. Maybe I'm missing something that already exists that would give me the adaptive throttling. We cannot make do with the static throttling (sync_speed_max and min) since that would be too difficult to get right for varying IO throughputs form the different RAID1 replicas. >>> >>> Thanks >>> ~ Hari >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> >> >> >> -- >> Roberto Spadim >> Spadim Technology / SPAEmpresarial >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > > > -- > Roberto Spadim > Spadim Technology / SPAEmpresarial > -- > To unsubscribe from this list: send the line "unsubscribe linux-raid" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
