On 5/30/12 7:41 AM, Stefan Priebe - Profihost AG wrote:
Hi Mark,
didn't had the time to answer your mails - but i will get on this one first.
Would you mind installing blktrace and running "blktrace -o test-3.4 -d
/dev/sdb" on the OSD node during a short (say 60s) test on 3.4?
sure no problem.
here it is:
http://www.mediafire.com/?6cw87btn7mzco25
Output:
=== sdb ===
CPU 0: 18075 events, 848 KiB data
CPU 1: 10738 events, 504 KiB data
CPU 2: 8639 events, 405 KiB data
CPU 3: 8614 events, 404 KiB data
CPU 4: 0 events, 0 KiB data
CPU 5: 0 events, 0 KiB data
CPU 6: 143 events, 7 KiB data
CPU 7: 0 events, 0 KiB data
Total: 46209 events (dropped 0), 2167 KiB data
If you could archive/send me the results, that might help us get an idea
of what is actually getting sent out to the disk. Your data disk
throughput on 3.0 looks pretty close to what I normally get (including
on 3.4). I'm guessing the issue you are seeing on 3.4 is probably not
the seek problem I mentioned earlier (unless something is causing so
many seeks that it more or less paralyzes the disk).
As i have a SSD i can't believe seeks can be a problem.
Stefan
Ok, I put up a seekwatcher movie showing the writes going to your SSD:
http://nhm.ceph.com/movies/mailinglist-tests/stefan.mpg
Some quick observations:
In your blktrace results there are some really big gaps after cfq
schedule dispatch:
8,16 0 0 11.386025866 0 m N cfq schedule dispatch
8,16 2 975 12.393446988 3074 A WS 176147976 + 8 <-
(8,17) 176145928
8,16 0 0 12.762164080 0 m N cfq schedule dispatch
8,16 0 2193 13.355165118 3312 A WSM 175875008 + 227 <-
(8,17) 175872960
Specifically, the gap in the movie where there is no write activity
around second 30 correlates in the blktrace results with one of these
stalls:
8,16 0 0 29.548567957 0 m N cfq schedule dispatch
8,16 2 2185 34.548923918 2688 A W 2192 + 8 <- (8,17) 144
As to why this is happening, I don't know yet. I'll have more later.
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
