Hi, We added some new OSDs today and since we've recently written many many (small/tiny) objects to a test pool, backfilling those new disks is going to take something like 24hrs. I'm therefore curious if we can speed up the recovery at all or if the default settings in cuttlefish already bring us to the limit. Here is a line from the mons earlier today just after the OSDs were started: 2013-07-18 21:05:38.017063 mon.0 128.142.142.156:6789/0 27124 : [INF] pgmap v111472: 9464 pgs: 8115 active+clean, 862 active+remapped+wait_backfill, 3 active+recovery_wait, 484 active+remapped+backfilling; 15994 GB data, 55536 GB used, 1 380 TB / 1434 TB avail; 12220476/151694232 degraded (8.056%); recovering 254 o/s, 93913KB/s There are 48 new OSDs, so the defaults of 10 max backfills roughly corresponds with the 484 backfilling. I started injecting higher backfilling options, specifically osd max backfills = 20 osd recovery max active = 10 and that gives me something like this: 2013-07-18 21:22:56.546094 mon.0 128.142.142.156:6789/0 27984 : [INF] pgmap v112308: 9464 pgs: 8129 active+clean, 398 active+remapped+wait_backfill, 3 active+recovery_wait, 933 active+remapped+backfilling, 1 active+clean+scrubbing; 15994 GB data, 55567 GB used, 1380 TB / 1434 TB avail; 11982626/151538728 degraded (7.907%); recovering 299 o/s, 114MB/s but immediately I start to see slow requests piling up. Trying with the different combinations, I found that it's the "max active = 10" setting that leads to the slow requests. With a 20/5 setting, there are no slow requests, but the recovery rate doesn't increase anyway. So I'm wondering if you all agree that this indicates that the 10/5 setting for backfill/max active is already the limit for our cluster, at least with this current set of test objects we have? Or am I missing another option that should be tweaked to get more recovery throughput? Thanks in advance, Dan -- Dan van der Ster CERN IT-DSS _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |