I've tried a variety of settings... ranging from 17 to 32768. Yes.. with stripe_cache_size set to 17, I see a C/T of rmw's. And my TP goes in the toilet -- even with the RAM disks, I get only about 30M/s. Bob Kierski Senior Storage Performance Engineer Cray Inc. 380 Jackson Street Suite 210 St. Paul, MN 55101 Tele: 651-967-9590 Fax: 651-605-9001 Cell: 651-890-7461 -----Original Message----- From: Phil Turmel [mailto:philip@xxxxxxxxxx] Sent: Wednesday, December 02, 2015 9:37 AM To: Robert Kierski; Dallas Clement Cc: linux-raid@xxxxxxxxxxxxxxx Subject: Re: RAID 5,6 sequential writing seems slower in newer kernels On 12/02/2015 10:28 AM, Robert Kierski wrote: > Thanks for the response. > > Nice try... But, the reason I’m using the 3.18.4 kernel is that it has the parallelization. I've got group_thread_cnt set to 32. I'm watching the CPU's with mpstat, and they're pretty much idle. I'm also watching the system traces with perf. It claims that only 11.9% of my time is spent doing the xor. Hmm. Ok. > I've got my CS set at 128k. I have noticed that if I set the CS to 32k, the TP is about 2x. I'm pretty sure the problem is that the 1M writes I'm doing are being broken into 4K pages, and then reassembled before going to disk. I think you're right. What is your stripe cache size? > Also, this is independent of the IO Scheduler. I've tried all 3 and got the same results. If your stripe cache is too small, sequential writes with large chunks can exhaust the cache before complete stripes are written, turning all of those partial stripe writes into read-modify-write cycles. ��.n��������+%������w��{.n�����{����w��ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
