Hi, sorry for the delay - productional system unfortunately ;-( Am 04.02.2016 um 15:38 schrieb Yan, Zheng: >> On Feb 4, 2016, at 17:00, Michael Metz-Martini | SpeedPartner GmbH <metz@xxxxxxxxxxxxxxx> wrote: >> Am 04.02.2016 um 09:43 schrieb Yan, Zheng: >>> On Thu, Feb 4, 2016 at 4:36 PM, Michael Metz-Martini | SpeedPartner >>> GmbH <metz@xxxxxxxxxxxxxxx> wrote: >>>> Am 03.02.2016 um 15:55 schrieb Yan, Zheng: >>>>>> On Feb 3, 2016, at 21:50, Michael Metz-Martini | SpeedPartner GmbH <metz@xxxxxxxxxxxxxxx> wrote: >>>>>> Am 03.02.2016 um 12:11 schrieb Yan, Zheng: >>>>>>>> On Feb 3, 2016, at 17:39, Michael Metz-Martini | SpeedPartner GmbH <metz@xxxxxxxxxxxxxxx> wrote: >>>>>>>> Am 03.02.2016 um 10:26 schrieb Gregory Farnum: >>>>>>>>> On Tue, Feb 2, 2016 at 10:09 PM, Michael Metz-Martini | SpeedPartner >>>>>> 2016-02-03 14:42:25.581840 7fadfd280700 0 log_channel(default) log >>>>>> [WRN] : 7 slow requests, 6 included below; oldest blocked for > >>>>>> 62.125785 secs >>>>>> 2016-02-03 14:42:25.581849 7fadfd280700 0 log_channel(default) log >>>>>> [WRN] : slow request 62.125785 seconds old, received at 2016-02-03 >>>>>> 14:41:23.455812: client_request(client.10199855:1313157 getattr >>>>>> pAsLsXsFs #100815bd349 2016-02-03 14:41:23.452386) currently failed to >>>>>> rdlock, waiting >>>>> This seems like dirty page writeback is too slow. Is there any hung OSD request in /sys/kernel/debug/ceph/xxx/osdc? >> Got it. http://www.michael-metz.de/osdc.txt.gz (about 500kb uncompressed) > That’s quite a lot requests. Could you pick some requests in osdc, and check how long do these requests last. After stopping load/access to cephfs there are a few requests left: 330 osd87 5.72c3bf71 100826d5cdc.00000002 write 508 osd87 5.569ad068 100826d5d18.00000000 write 668 osd87 5.3db54b00 100826d5d4d.00000001 write 799 osd87 5.65f8c4e0 100826d5d79.00000000 write 874 osd87 5.d238da71 100826d5d98.00000000 write 1023 osd87 5.705950e0 100826d5e2d.00000000 write 1277 osd87 5.33673f71 100826d5f2a.00000000 write 1329 osd87 5.e81ab868 100826d5f5e.00000000 write 1392 osd87 5.aea1c771 100826d5f9c.00000000 write osd.87 is near full and currently has some pg's with backfill_toofull but can this be the reason for this? -- Kind regards Michael Metz-Martini _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |