Dallas,
The threshold between iodepth=32 and iodepth=64 might be caused by exceeding the stripe cache size.
In my opinion, if you're writing chunk aligned data, you shouldn't be doing any RMW's. That you're doing small numbers of RMW's with small iodepth, indicates that you're using the stripe cache.
I've tried similar tests where I've set the stripe cache size to 17 (the smallest you can set it to), and then did perfectly aligned IO's. The results showed that I was doing massive amounts of RMW, and my performance was horrible.
Your FIO job file looks ordinary (and by that I mean.... "good"). While I wouldn't have picked bs=1408k, because it's aligned, I wouldn't expect that to be a cause for problem.
You should be able to set iodepth to whatever you want. You could set it to a billion. The OS should block additional requests until the underlying device's queue has available space. Iodepth shouldn't affect RMW when you're doing aligned writes. In my opnion, increasing iodepth should only help... not hurt. If it causes low memory, that could be an issue, but it shouldn't increase the amount of RMW.
Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax: 651-605-9001
Cell: 651-890-7461
��.n��������+%������w��{.n�����{����w��ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
