But yes, I will try to increase OSD verbosity. Le jeudi 17 septembre 2015 à 20:28 -0700, GuangYang a écrit : > Which version are you using? > > My guess is that the request (op) is waiting for lock (might be > ondisk_read_lock of the object, but a debug_osd=20 should be helpful > to tell what happened to the op). > > How do you tell the IO wait is near to 0 (by top?)? > > Thanks, > Guang > ---------------------------------------- > > From: ceph.list@xxxxxxxxx > > To: ceph-users@xxxxxxxxxxxxxx > > Date: Fri, 18 Sep 2015 02:43:49 +0200 > > Subject: Re: Lot of blocked operations > > > > Some additionnal informations : > > - I have 4 SSD per node. > > - the CPU usage is near 0 > > - IO wait is near 0 too > > - bandwith usage is also near 0 > > > > The whole cluster seems waiting for something... but I don't see > > what. > > > > > > Le vendredi 18 septembre 2015 à 02:35 +0200, Olivier Bonvalet a > > écrit : > > > Hi, > > > > > > I have a cluster with lot of blocked operations each time I try > > > to > > > move > > > data (by reweighting a little an OSD). > > > > > > It's a full SSD cluster, with 10GbE network. > > > > > > In logs, when I have blocked OSD, on the main OSD I can see that > > > : > > > 2015-09-18 01:55:16.981396 7f89e8cb8700 0 log [WRN] : 2 slow > > > requests, 1 included below; oldest blocked for> 33.976680 secs > > > 2015-09-18 01:55:16.981402 7f89e8cb8700 0 log [WRN] : slow > > > request > > > 30.125556 seconds old, received at 2015-09-18 01:54:46.855821: > > > osd_op(client.29760717.1:18680817544 > > > rb.0.1c16005.238e1f29.00000000027f [write 180224~16384] > > > 6.c11916a4 > > > snapc 11065=[11065,10fe7,10f69] ondisk+write e845819) v4 > > > currently > > > reached pg > > > 2015-09-18 01:55:46.986319 7f89e8cb8700 0 log [WRN] : 2 slow > > > requests, 1 included below; oldest blocked for> 63.981596 secs > > > 2015-09-18 01:55:46.986324 7f89e8cb8700 0 log [WRN] : slow > > > request > > > 60.130472 seconds old, received at 2015-09-18 01:54:46.855821: > > > osd_op(client.29760717.1:18680817544 > > > rb.0.1c16005.238e1f29.00000000027f [write 180224~16384] > > > 6.c11916a4 > > > snapc 11065=[11065,10fe7,10f69] ondisk+write e845819) v4 > > > currently > > > reached pg > > > > > > How should I read that ? What this OSD is waiting for ? > > > > > > Thanks for any help, > > > > > > Olivier > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@xxxxxxxxxxxxxx > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |