Hello, Using 5c65e1ee3932a021cfd900a74cdc1d43b9103f0f with large amount of data commit and relatively low PG rate, I`ve observed unexplainable long recovery times for PGs even if the degraded object count is almost zero: 04:44:42.521896 mon.0 [INF] pgmap v24807947: 2048 pgs: 911 active+clean, 1131 active+recovery_wait, 6 active+recovering; 5389 GB data, 16455 GB used, 87692 GB / 101 TB avail; 5839KB/s rd, 2986KB/s wr, 567op/s; 865/4162926 degraded (0.021%); recovering 2 o/s, 9251KB/s at this moment we have freshly restarted cluster and large amount of PGs in recovery_wait state; after a couple of minutes picture changes a little: 2013-11-13 05:30:18.020093 mon.0 [INF] pgmap v24809483: 2048 pgs: 939 active+clean, 1105 active+recovery_wait, 4 active+recovering; 5394 GB data, 16472 GB used, 87676 GB / 101 TB avail; 1627KB/s rd, 3866KB/s wr, 1499op/s; 2456/4167201 degraded (0.059%) and after a couple of hours we`re reaching a peak by degraded objects, PGs still moving to active+clean: 2013-11-13 10:05:36.245917 mon.0 [INF] pgmap v24816326: 2048 pgs: 1191 active+clean, 854 active+recovery_wait, 3 active+recovering; 5467 GB data, 16690 GB used, 87457 GB / 101 TB avail; 16339KB/s rd, 18006KB/s wr, 16025op/s; 23495/4223061 degraded (0.556%) After peak was passed, object count starts to decrease and seems cluster will reach completely clean state in next ten hours. For example, with PG count ten times higher recovery goes way faster: 2013-11-05 03:20:56.330767 mon.0 [INF] pgmap v24143721: 27648 pgs: 26171 active+clean, 1474 active+recovery_wait, 3 active+recovering; 7855 GB data, 25609 GB used, 78538 GB / 101 TB avail; 3 298KB/s rd, 7746KB/s wr, 3581op/s; 183/6554634 degraded (0.003%) 2013-11-05 04:04:55.779345 mon.0 [INF] pgmap v24145291: 27648 pgs: 27646 active+clean, 1 active+recovery_wait, 1 active+recovering; 7857 GB data, 25615 GB used, 78533 GB / 101 TB avail; 999KB/s rd, 690KB/s wr, 563op/s Recovery and backfill settings was the same during all tests: "osd_max_backfills": "1", "osd_backfill_full_ratio": "0.85", "osd_backfill_retry_interval": "10", "osd_recovery_threads": "1", "osd_recover_clone_overlap": "true", "osd_backfill_scan_min": "64", "osd_backfill_scan_max": "512", "osd_recovery_thread_timeout": "30", "osd_recovery_delay_start": "300", "osd_recovery_max_active": "5", "osd_recovery_max_chunk": "8388608", "osd_recovery_forget_lost_objects": "false", "osd_kill_backfill_at": "0", "osd_debug_skip_full_check_in_backfill_reservation": "false", "osd_recovery_op_priority": "10", Also during recovery some heartbeats may miss, it is not related to the current situation but observed for a very long time(for now, seems four-seconds delays between heartbeats distributed almost randomly over a time flow): 2013-11-13 14:57:11.316459 mon.0 [INF] pgmap v24826822: 2048 pgs: 1513 active+clean, 529 active+recovery_wait, 6 active+recovering; 5469 GB data, 16708 GB used, 87440 GB / 101 TB avail; 16098KB/s rd, 4085KB/s wr, 623op/s; 15670/4227330 degraded (0.371%) 2013-11-13 14:57:12.328538 mon.0 [INF] pgmap v24826823: 2048 pgs: 1513 active+clean, 529 active+recovery_wait, 6 active+recovering; 5469 GB data, 16708 GB used, 87440 GB / 101 TB avail; 3806KB/s rd, 3446KB/s wr, 284op/s; 15670/4227330 degraded (0.371%) 2013-11-13 14:57:13.336618 mon.0 [INF] pgmap v24826824: 2048 pgs: 1513 active+clean, 529 active+recovery_wait, 6 active+recovering; 5469 GB data, 16708 GB used, 87440 GB / 101 TB avail; 11051KB/s rd, 12171KB/s wr, 1470op/s; 15670/4227330 degraded (0.371%) 2013-11-13 14:57:16.317271 mon.0 [INF] pgmap v24826825: 2048 pgs: 1513 active+clean, 529 active+recovery_wait, 6 active+recovering; 5469 GB data, 16708 GB used, 87440 GB / 101 TB avail; 3610KB/s rd, 3171KB/s wr, 1820op/s; 15670/4227330 degraded (0.371%) 2013-11-13 14:57:17.366554 mon.0 [INF] pgmap v24826826: 2048 pgs: 1513 active+clean, 529 active+recovery_wait, 6 active+recovering; 5469 GB data, 16708 GB used, 87440 GB / 101 TB avail; 11323KB/s rd, 1759KB/s wr, 13195op/s; 15670/4227330 degraded (0.371%) 2013-11-13 14:57:18.379340 mon.0 [INF] pgmap v24826827: 2048 pgs: 1513 active+clean, 529 active+recovery_wait, 6 active+recovering; 5469 GB data, 16708 GB used, 87440 GB / 101 TB avail; 38113KB/s rd, 7461KB/s wr, 46511op/s; 15670/4227330 degraded (0.371%) _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |