Hi guys, I'm testing rbd with fio. I have test it for more than 3 hours, and then it happened to see that fio showing instant iops is 0. then I catch the trace of rbd with blktrace, and found that the kernel worker thread "kworker" is busy with writing meta data to rbd with the flag "WM". I think this may be the cause of fio showing 0 iops. can anyone help me to analyse what the kworker thread is doing now, and how can we avoid or reduce the happening of this situation? I have attach the blktrace output beblow after parsing. Hope to get your suggestions. Best Regards 252,0 2 0 0.121945547 0 m N cfq167A / complete rqnoidle 0 252,0 2 0 0.121946391 0 m N cfq167A / set_slice=40 252,0 2 8 0.123667363 10264 C WS 1636200920 + 8 [0] 252,0 2 0 0.123675445 0 m N cfq6788SN / complete rqnoidle 1 252,0 2 0 0.123675975 0 m N cfq schedule dispatch 252,0 2 0 0.123685200 0 m N cfq167A / fifo=ffff8800641b2aa0 252,0 2 0 0.123685739 0 m N cfq167A / dispatch_insert 252,0 2 0 0.123687302 0 m N cfq167A / dispatched a request 252,0 2 0 0.123687796 0 m N cfq167A / activate rq, drv=1 252,0 2 9 0.123688133 246 D WM 385944760 + 8 [kworker/2:1H] 252,0 2 10 0.129434563 10264 C WM 385944760 + 8 [0] 252,0 2 0 0.129442292 0 m N cfq167A / complete rqnoidle 0 252,0 2 0 0.129442652 0 m N cfq schedule dispatch 252,0 2 0 0.129452994 0 m N cfq167A / dispatch_insert 252,0 2 0 0.129454098 0 m N cfq167A / dispatched a request 252,0 2 0 0.129454503 0 m N cfq167A / activate rq, drv=1 252,0 2 11 0.129454845 246 D WM 385944792 + 16 [kworker/2:1H] 252,0 1 1 0.129465968 7519 G WM 393396232 + 8 [kworker/u8:0] 252,0 1 2 0.129466924 7519 P N [kworker/u8:0] 252,0 1 3 0.129473599 7519 Q WM 393396304 + 8 [kworker/u8:0] 252,0 1 4 0.129476470 7519 G WM 393396304 + 8 [kworker/u8:0] 252,0 1 5 0.129480790 7519 Q WM 393396328 + 8 [kworker/u8:0] 252,0 1 6 0.129481837 7519 G WM 393396328 + 8 [kworker/u8:0] 252,0 1 7 0.129484096 7519 Q WM 393396416 + 8 [kworker/u8:0] 252,0 1 8 0.129485170 7519 G WM 393396416 + 8 [kworker/u8:0] 252,0 1 9 0.129486519 7519 Q WM 393396432 + 8 [kworker/u8:0] 252,0 1 10 0.129487356 7519 G WM 393396432 + 8 [kworker/u8:0] 252,0 1 11 0.129489915 7519 Q WM 393396536 + 8 [kworker/u8:0] 252,0 1 12 0.129490942 7519 G WM 393396536 + 8 [kworker/u8:0] 252,0 1 13 0.129491905 7519 Q WM 393396544 + 8 [kworker/u8:0] 252,0 1 14 0.129492514 7519 M WM 393396544 + 8 [kworker/u8:0] 252,0 1 15 0.129493756 7519 Q WM 393396592 + 8 [kworker/u8:0] 252,0 1 16 0.129494196 7519 G WM 393396592 + 8 [kworker/u8:0] 252,0 1 17 0.129496691 7519 Q WM 393396664 + 8 [kworker/u8:0] 252,0 1 18 0.129497171 7519 G WM 393396664 + 8 [kworker/u8:0] 252,0 1 19 0.129500082 7519 Q WM 393396744 + 8 [kworker/u8:0] 252,0 1 20 0.129500597 7519 G WM 393396744 + 8 [kworker/u8:0] 252,0 1 21 0.129501234 7519 Q WM 393396776 + 8 [kworker/u8:0] 252,0 1 22 0.129501987 7519 G WM 393396776 + 8 [kworker/u8:0] 252,0 1 23 0.129505341 7519 Q WM 393396888 + 8 [kworker/u8:0] 252,0 1 24 0.129505791 7519 G WM 393396888 + 8 [kworker/u8:0] 252,0 1 25 0.129506418 7519 Q WM 393396896 + 8 [kworker/u8:0] 252,0 1 26 0.129506522 7519 M WM 393396896 + 8 [kworker/u8:0] 252,0 1 27 0.129507455 7519 Q WM 393396928 + 8 [kworker/u8:0] 252,0 1 28 0.129508168 7519 G WM 393396928 + 8 [kworker/u8:0] 252,0 1 29 0.129510621 7519 Q WM 393396968 + 8 [kworker/u8:0] 252,0 1 30 0.129511363 7519 G WM 393396968 + 8 [kworker/u8:0] 252,0 1 31 0.129511981 7519 Q WM 393396976 + 8 [kworker/u8:0] 252,0 1 32 0.129512076 7519 M WM 393396976 + 8 [kworker/u8:0] 252,0 1 33 0.129513837 7519 Q WM 393397056 + 8 [kworker/u8:0] 252,0 1 34 0.129514422 7519 G WM 393397056 + 8 [kworker/u8:0] 252,0 1 35 0.129515900 7519 Q WM 393397088 + 8 [kworker/u8:0] 252,0 1 36 0.129516358 7519 G WM 393397088 + 8 [kworker/u8:0] 252,0 1 37 0.129519459 7519 Q WM 393397216 + 8 [kworker/u8:0] 252,0 1 38 0.129519963 7519 G WM 393397216 + 8 [kworker/u8:0] 252,0 1 39 0.129520590 7519 Q WM 393397240 + 8 [kworker/u8:0] 252,0 1 40 0.129521228 7519 G WM 393397240 + 8 [kworker/u8:0] 252,0 1 41 0.129522524 7519 I WM 393396232 + 8 [kworker/u8:0] 252,0 1 0 0.129523670 0 m N cfq167A / insert_request 252,0 1 42 0.129524528 7519 I WM 393396304 + 8 [kworker/u8:0] 252,0 1 0 0.129524864 0 m N cfq167A / insert_request 252,0 1 43 0.129525202 7519 I WM 393396328 + 8 [kworker/u8:0] 252,0 1 0 0.129525568 0 m N cfq167A / insert_request 252,0 1 44 0.129525740 7519 I WM 393396416 + 8 [kworker/u8:0] 252,0 1 0 0.129526052 0 m N cfq167A / insert_request 252,0 1 45 0.129526235 7519 I WM 393396432 + 8 [kworker/u8:0] 252,0 1 0 0.129526497 0 m N cfq167A / insert_request 252,0 1 46 0.129526666 7519 I WM 393396536 + 16 [kworker/u8:0] 252,0 1 0 0.129526916 0 m N cfq167A / insert_request 252,0 1 47 0.129527127 7519 I WM 393396592 + 8 [kworker/u8:0] 252,0 1 0 0.129527359 0 m N cfq167A / insert_request 252,0 1 48 0.129527525 7519 I WM 393396664 + 8 [kworker/u8:0] 252,0 1 0 0.129527736 0 m N cfq167A / insert_request 252,0 1 49 0.129528034 7519 I WM 393396744 + 8 [kworker/u8:0] 252,0 1 0 0.129528277 0 m N cfq167A / insert_request 252,0 1 50 0.129528445 7519 I WM 393396776 + 8 [kworker/u8:0] 252,0 1 0 0.129528700 0 m N cfq167A / insert_request 252,0 1 51 0.129528878 7519 I WM 393396888 + 16 [kworker/u8:0] 252,0 1 0 0.129529120 0 m N cfq167A / insert_request 252,0 1 52 0.129529293 7519 I WM 393396928 + 8 [kworker/u8:0] 252,0 1 0 0.129529534 0 m N cfq167A / insert_request 252,0 1 53 0.129529728 7519 I WM 393396968 + 16 [kworker/u8:0] 252,0 1 0 0.129529971 0 m N cfq167A / insert_request 252,0 1 54 0.129530139 7519 I WM 393397056 + 8 [kworker/u8:0] 252,0 1 0 0.129530368 0 m N cfq167A / insert_request 252,0 1 55 0.129530545 7519 I WM 393397088 + 8 [kworker/u8:0] 252,0 1 0 0.129530798 0 m N cfq167A / insert_request 252,0 1 56 0.129530963 7519 I WM 393397216 + 8 [kworker/u8:0] 252,0 1 0 0.129531201 0 m N cfq167A / insert_request 252,0 1 57 0.129531571 7519 U N [kworker/u8:0] 16 252,0 1 58 0.129531932 7519 P N [kworker/u8:0] 252,0 1 59 0.129532608 7519 Q WM 393397248 + 8 [kworker/u8:0] 252,0 1 60 0.129532741 7519 M WM 393397248 + 8 [kworker/u8:0] 252,0 1 61 0.129536392 7519 Q WM 393397392 + 8 [kworker/u8:0] 252,0 1 62 0.129536857 7519 G WM 393397392 + 8 [kworker/u8:0] 252,0 1 63 0.129538102 7519 Q WM 393397440 + 8 [kworker/u8:0] 252,0 1 64 0.129538494 7519 G WM 393397440 + 8 [kworker/u8:0] 252,0 1 65 0.129540776 7519 Q WM 393397544 + 8 [kworker/u8:0] 252,0 1 66 0.129541420 7519 G WM 393397544 + 8 [kworker/u8:0] 252,0 1 67 0.129542032 7519 Q WM 393397552 + 8 [kworker/u8:0] 252,0 1 68 0.129542125 7519 M WM 393397552 + 8 [kworker/u8:0] 252,0 1 69 0.129542777 7519 Q WM 393397592 + 8 [kworker/u8:0] 252,0 1 70 0.129543292 7519 G WM 393397592 + 8 [kworker/u8:0] 252,0 1 71 0.129543897 7519 Q WM 393397616 + 8 [kworker/u8:0] 252,0 1 72 0.129544354 7519 G WM 393397616 + 8 [kworker/u8:0] 252,0 1 73 0.129544998 7519 Q WM 393397640 + 8 [kworker/u8:0] 252,0 1 74 0.129545439 7519 G WM 393397640 + 8 [kworker/u8:0] 252,0 1 75 0.129549160 7519 Q WM 393397776 + 8 [kworker/u8:0] 252,0 1 76 0.129549651 7519 G WM 393397776 + 8 [kworker/u8:0] 252,0 1 77 0.129550869 7519 Q WM 393397816 + 8 [kworker/u8:0] 252,0 1 78 0.129551504 7519 G WM 393397816 + 8 [kworker/u8:0] 252,0 1 79 0.129552131 7519 Q WM 393397824 + 8 [kworker/u8:0] 252,0 1 80 0.129552227 7519 M WM 393397824 + 8 [kworker/u8:0] 252,0 1 81 0.129554366 7519 Q WM 393397912 + 8 [kworker/u8:0] 252,0 1 82 0.129555187 7519 G WM 393397912 + 8 [kworker/u8:0] 252,0 1 83 0.129555861 7519 Q WM 393397920 + 8 [kworker/u8:0] 252,0 1 84 0.129555955 7519 M WM 393397920 + 8 [kworker/u8:0] -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140801/c19ce3ec/attachment.htm>
 |