Help solving a performance mystery would be welcome.
I have two test programs:
* one totally CPU bound (it just infinitely loops)
* the other I/O bound (it loops, each time doing
* a write,
* an fsync or fdatasync (I get similar behavior with either),
* an lseek to back up to the same position again
Under ext2, if I run these two together, they each run nearly as fast
as they would alone, since when the disk completes a write, the I/O
bound process has been sleeping long enough to have gained priority
over the CPU bound process, preempts for long enough to submit the next
write request, and then goes back to sleep, letting the CPU bound one
run while the disk does its thing.
Under ext3, I get a totally different, and very mysterious, behavior.
Sometimes the I/O bound process will make very low rates of progress
for tens of seconds. For example, this is evident in the following
"vmstat 1" output. I start with just the I/O bound process running.
Then at some point I start the CPU bound process, and the "bo" column
drops precipitously. After tens of seconds, the "bo" column goes back
up again, for no apparent reason. How long this takes to happens
varies considerably. Sometimes the I/O rate never dips in the first
place, or at most only very briefly.
Can anyone explain this phenomenon? I'm wondering if it has something
to do with the need to schedule two different threads -- my I/O-bound
benchmark and the kjournald. But I can't think why either of them
wouldn't gain priority.
max 30% vmstat 1
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 11064 2316 27888 30260 0 3 22 816 788 1441 34 5 61
3 1 1 11196 2332 27936 30224 0 132 0 1156 480 948 93 7 0
1 1 1 11196 2324 28008 30152 0 0 0 1100 346 945 97 3 0
2 0 1 11196 2320 27956 30208 0 0 0 1120 348 772 97 3 0
2 1 1 11196 2304 28032 30144 0 0 0 1160 348 716 99 1 0
1 1 1 11196 2304 28048 30128 0 0 0 1116 366 728 99 1 0
1 1 1 11196 2304 28080 30096 0 0 0 1116 345 691 96 4 0
1 1 1 11196 2304 28000 30176 0 0 0 1148 350 717 98 2 0
1 1 1 11196 2304 28028 30148 0 0 0 1132 342 707 98 2 0
0 1 1 11196 2356 27904 30272 0 0 0 1136 351 758 39 3 58
1 0 0 11196 2356 27904 30272 0 0 0 1152 356 758 1 3 96
2 0 0 11196 2336 27840 30304 0 0 0 820 287 585 34 2 64
2 0 0 11196 2332 27840 30304 0 0 0 92 146 192 99 1 0
2 0 0 11196 2332 27804 30340 0 0 0 76 154 198 100 0 0
2 0 0 11196 2332 27804 30340 0 0 0 64 142 176 99 1 0
2 0 0 11196 2332 27704 30440 0 0 0 64 147 180 100 0 0
2 0 0 11196 2332 27704 30440 0 0 0 80 147 179 100 0 0
4 0 0 11196 2332 27676 30468 0 0 0 64 145 176 99 1 0
3 0 0 11196 2336 27676 30468 0 0 0 64 145 187 99 1 0
2 0 0 11196 2336 27648 30496 0 0 0 72 141 180 100 0 0
2 0 0 11196 2336 27648 30496 0 0 0 68 143 186 99 1 0
4 0 0 11196 2332 27464 30680 0 0 0 64 142 168 100 0 0
3 0 0 11196 2332 27464 30680 0 0 0 72 144 181 100 0 0
3 0 0 11196 2332 27420 30724 0 0 0 64 141 181 99 1 0
2 0 0 11196 2324 27428 30724 0 0 0 64 141 175 99 1 0
2 0 0 11196 2324 27432 30720 0 0 0 72 146 189 99 1 0
3 0 0 11196 2324 27440 30712 0 0 0 64 140 169 99 1 0
3 0 0 11196 2324 27204 30944 0 0 0 64 143 177 99 1 0
3 0 0 11196 2324 27204 30940 0 0 0 72 141 190 99 1 0
2 0 0 11196 2324 27068 31076 0 0 0 64 146 177 99 1 0
3 0 0 11196 2324 27068 31076 0 0 0 64 146 177 99 1 0
2 0 0 11196 2324 26972 31172 0 0 0 72 150 182 98 2 0
2 0 0 11196 2324 26972 31172 0 0 0 64 146 181 98 2 0
2 0 0 11196 2324 26852 31292 0 0 0 64 148 174 100 0 0
2 0 0 11196 2324 26700 31444 0 0 0 76 157 195 99 1 0
2 0 0 11196 2324 26700 31440 0 0 0 64 148 174 99 1 0
2 0 0 11196 2324 26276 31864 0 0 0 64 142 172 99 1 0
4 0 0 11196 2324 26284 31856 0 0 0 72 145 189 99 1 0
2 0 0 11196 2324 26256 31884 0 0 0 64 144 171 99 1 0
3 0 0 11196 2324 26256 31884 0 0 0 64 155 190 99 1 0
2 0 0 11196 2324 26256 31884 0 0 0 72 142 179 100 0 0
2 0 0 11196 2324 26256 31884 0 0 0 64 141 177 98 2 0
5 0 0 11196 2324 26276 31864 0 0 0 64 149 189 100 0 0
2 0 0 11196 2324 26292 31844 0 0 0 72 143 180 100 0 0
2 0 0 11196 2324 26308 31828 0 0 0 68 144 187 99 1 0
2 0 0 11196 2324 26320 31816 0 0 0 64 140 173 99 1 0
3 0 0 11196 2324 26332 31804 0 0 0 72 145 180 100 0 0
2 0 0 11196 2324 26336 31800 0 0 0 60 142 181 99 1 0
2 0 0 11196 2324 26340 31796 0 0 0 60 159 174 99 1 0
2 0 0 11196 2324 26352 31780 0 0 0 72 154 188 99 1 0
2 0 0 11196 2324 26364 31768 0 0 0 60 141 177 99 1 0
3 0 0 11196 2324 26372 31760 0 0 0 64 140 173 99 1 0
3 0 0 11196 2324 26376 31756 0 0 0 72 144 183 99 1 0
3 0 0 11196 2324 26392 31740 0 0 0 60 140 175 99 1 0
4 0 0 11196 2324 26400 31732 0 0 0 64 143 178 99 1 0
3 0 0 11196 2324 26404 31728 0 0 0 72 142 181 99 1 0
3 0 0 11196 2324 26424 31708 0 0 0 64 142 178 99 1 0
4 0 0 11196 2324 26432 31696 0 0 0 64 142 187 99 1 0
3 0 0 11196 2324 26460 31668 0 0 0 72 143 180 99 1 0
3 0 0 11196 2324 26480 31648 0 0 0 68 145 189 99 1 0
4 0 0 11196 2324 26496 31632 0 0 0 64 143 175 100 0 0
3 0 0 11196 2324 26512 31616 0 0 0 72 143 180 99 1 0
3 0 0 11196 2324 25848 32280 0 0 0 64 142 181 99 1 0
3 0 0 11196 2324 25852 32276 0 0 0 72 170 193 100 0 0
3 0 0 11196 2324 25852 32276 0 0 0 72 123 178 99 1 0
4 0 0 11196 2324 25852 32276 0 0 0 64 137 166 99 1 0
4 0 0 11196 2324 25868 32260 0 0 0 68 144 172 99 1 0
6 0 0 11196 2324 25876 32252 0 0 0 72 141 182 99 1 0
5 0 0 11196 2324 25900 32232 0 0 0 64 142 180 99 1 0
4 0 0 11196 2324 25920 32212 0 0 0 68 158 184 99 1 0
3 0 0 11196 2324 25924 32208 0 0 0 72 144 188 99 1 0
5 0 0 11196 2324 25932 32196 0 0 0 64 142 175 99 1 0
5 0 0 11196 2324 25952 32176 0 0 0 68 151 191 99 1 0
5 0 0 11196 2324 25972 32156 0 0 0 72 142 185 99 1 0
4 0 0 11196 2324 26000 32128 0 0 0 72 153 184 98 2 0
3 0 0 11196 2324 26012 32116 0 0 0 68 142 181 100 0 0
4 0 0 11196 2324 26028 32100 0 0 0 72 143 183 99 1 0
5 0 0 11196 2324 26048 32080 0 0 0 64 151 185 100 0 0
5 0 0 11196 2324 26064 32060 0 0 0 72 171 205 98 2 0
3 0 0 11196 2324 26096 32024 0 0 0 72 122 172 99 1 0
3 0 0 11196 2324 26116 32004 0 0 0 64 139 172 99 1 0
3 0 0 11196 2324 26128 31992 0 0 0 68 147 182 99 1 0
3 0 0 11196 2324 26136 31984 0 0 0 72 143 184 98 2 0
3 0 0 11196 2324 26148 31972 0 0 0 64 142 176 98 2 0
4 0 0 11196 2324 26156 31964 0 0 0 68 145 185 99 1 0
3 0 0 11196 2324 26156 31960 0 0 0 72 147 174 99 1 0
3 0 0 11196 2324 26156 31960 0 0 0 64 142 183 99 1 0
4 0 0 11196 2324 26156 31960 0 0 0 68 142 177 99 1 0
3 0 0 11196 2324 26172 31944 0 0 0 72 150 178 98 2 0
4 0 0 11196 2324 26180 31936 0 0 0 64 145 179 97 3 0
4 0 0 11196 2324 26196 31852 0 0 324 68 177 216 99 1 0
2 1 1 11196 2388 25732 31980 24 0 212 652 269 490 99 1 0
2 1 1 11196 2360 25960 31780 0 0 0 1060 332 653 100 0 0
3 0 1 11192 2660 26252 31460 0 0 0 1164 350 739 97 3 0
1 1 1 11192 2568 26488 31316 0 0 0 1172 350 725 99 1 0
1 1 1 11192 2424 26020 31928 0 0 0 1176 359 724 98 2 0
2 1 1 11192 2364 26324 31684 0 0 0 1116 346 704 97 3 0
1 1 1 11192 2364 26584 31424 0 0 0 1116 340 692 98 2 0
2 1 1 11192 2364 26924 31084 0 0 0 1116 541 1280 94 6 0
2 1 1 11192 2364 27192 30816 0 0 0 1148 459 1119 97 3 0
_______________________________________________
Ext3-users@redhat.com
https://listman.redhat.com/mailman/listinfo/ext3-users
