I found a problem with the vanilla kernel from 2.6.22 (include 23, 24): the situation is to test a customized linux distribution under heavy IO load: 1. the client process initiates tens of POSIX threads (using libpthread), each thread uses aio_write or aio_read(using librt) operating on one small file, then close it and write another small file; the whole objective is to get a maxium throughput of small files, by generating heavy aio stress on the system; I have tested the vanilla kernel 2.6.22/23/24.y, all these kernels have the common problem: 1. I use top, vmstat, and iostat to monitor the system performance, I found that the iowait time of CPU is high at most time, above 60%, and not stable while the client process running, although the throughput is stable, the CPU iowait time not stable; As a result of instability processes felt too long and unacceptable delay. 2. in seldom testing cases, the system even stopped: with bi/bo are 0 and iowait 100%, all writing processes are blocking with uninterruptible state (D state in ps output), these cases are all system running after several days writing, but cannot guarantee to reproduce; First I suspect the filesystem or the storage medium have problems, I tried ext2/ext3/reiserfs/xfs, for different filesytems and SATA/SAS, IDE disks, and several commercial hard RAID card for different storage medium, but the bad result remained; Then I tested 2.6.21.7 kernel, this results a stable iowait CPU time (below 10%) but not so good throughput of small files; Now I think the improvements of IO effciency in the development of 2.6.22 also caused the instability of iowait time, right? but how to find out? Simple bisecting seems not work. by the way, a question is how to guarantee a kernel not regress under heavy multi-thread aio writing load? ltp project seems not give the answer: http://ltp.sourceforge.net/tooltable.php -- Denis -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
