-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Are you testing with straw or straw2? - ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 13, 2015 at 2:22 AM, Xusangdi wrote: > Hi Sage, > > Recently when I was learning about the crush rules I noticed that the step chooseleaf may cause some unnecessary pg migrations when OSDs are outed. > For example, for a cluster of 4 hosts with 2 OSDs each, after host1(osd.2, osd.3) is down, the mapping differences would be like this: > pgid before <-> after diff diff_num > 0.1e [5, 1, 2] <-> [5, 1, 7] [2] 1 > 0.1f [0, 7, 3] <-> [0, 7, 4] [3] 1 > 0.1a [0, 4, 3] <-> [0, 4, 6] [3] 1 > 0.5 [6, 3, 1] <-> [6, 0, 5] [1, 3] 2 > 0.4 [5, 6, 2] <-> [5, 6, 0] [2] 1 > 0.7 [3, 7, 0] <-> [7, 0, 4] [3] 1 > 0.6 [2, 1, 7] <-> [0, 7, 4] [1, 2] 2 > 0.9 [3, 4, 0] <-> [5, 0, 7] [3, 4] 2 > 0.15 [2, 6, 1] <-> [6, 0, 5] [1, 2] 2 > 0.14 [3, 6, 5] <-> [7, 4, 1] [3, 5, 6] 3 > 0.17 [0, 5, 2] <-> [0, 5, 6] [2] 1 > 0.16 [0, 4, 2] <-> [0, 4, 7] [2] 1 > 0.11 [4, 7, 2] <-> [4, 7, 1] [2] 1 > 0.10 [0, 3, 6] <-> [0, 7, 4] [3, 6] 2 > 0.13 [1, 7, 3] <-> [1, 7, 4] [3] 1 > 0.a [0, 2, 7] <-> [0, 7, 4] [2] 1 > 0.c [5, 0, 3] <-> [5, 0, 6] [3] 1 > 0.b [2, 5, 7] <-> [4, 7, 0] [2, 5] 2 > 0.18 [7, 2, 4] <-> [7, 4, 0] [2] 1 > 0.f [2, 7, 5] <-> [6, 4, 0] [2, 5, 7] 3 > Changed pg ratio: 30 / 32 > > I tried to change the code (please see https://github.com/ceph/ceph/pull/6242) and after the modification the result would be like this: > pgid before <-> after diff diff_num > 0.1e [5, 0, 3] <-> [5, 0, 7] [3] 1 > 0.1f [0, 6, 3] <-> [0, 6, 4] [3] 1 > 0.1a [0, 5, 2] <-> [0, 5, 6] [2] 1 > 0.5 [6, 3, 0] <-> [6, 0, 5] [3] 1 > 0.4 [5, 7, 2] <-> [5, 7, 0] [2] 1 > 0.7 [3, 7, 1] <-> [7, 1, 5] [3] 1 > 0.6 [2, 0, 7] <-> [0, 7, 4] [2] 1 > 0.9 [3, 5, 1] <-> [5, 1, 7] [3] 1 > 0.15 [2, 6, 1] <-> [6, 1, 4] [2] 1 > 0.14 [3, 7, 5] <-> [7, 5, 1] [3] 1 > 0.17 [0, 4, 3] <-> [0, 4, 6] [3] 1 > 0.16 [0, 4, 3] <-> [0, 4, 6] [3] 1 > 0.11 [4, 6, 3] <-> [4, 6, 0] [3] 1 > 0.10 [0, 3, 6] <-> [0, 6, 5] [3] 1 > 0.13 [1, 7, 3] <-> [1, 7, 5] [3] 1 > 0.a [0, 3, 6] <-> [0, 6, 5] [3] 1 > 0.c [5, 0, 3] <-> [5, 0, 6] [3] 1 > 0.b [2, 4, 6] <-> [4, 6, 1] [2] 1 > 0.18 [7, 3, 5] <-> [7, 5, 1] [3] 1 > 0.f [2, 6, 5] <-> [6, 5, 1] [2] 1 > Changed pg ratio: 20 / 32 > > Currently the only defect I can see from the change is that the chance for a given pg to successfully choose required available OSDs might be a bit lower compared with before. However, I believe it will cause problems only when the cluster is pretty small and degraded. And in that case, we can still make it workable by tuning some of the crushmap parameters such as chooseleaf_tries. > > Anyway I'm not sure if it would raise any other issues, could you please review it and maybe give me some suggestions? Thank you! > > ---------- > Best regards, > Sangdi > > ------------------------------------------------------------------------------------------------------------------------------------- > 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出 > 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 > 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 > 邮件！ > This e-mail and its attachments contain confidential information from H3C, which is > intended only for the person or entity whose address is listed above. Any use of the > information contained herein in any way (including, but not limited to, total or partial > disclosure, reproduction, or dissemination) by persons other than the intended > recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender > by phone or email immediately and delete it! -----BEGIN PGP SIGNATURE----- Version: Mailvelope v1.2.0 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWHRM4CRDmVDuy+mK58QAARVMP/jhhtyRsiUXw4kl2ikso F8CiAwPuGRMvFSa2CXqzvaHnNjiy8Q4uR8o0KgcR04eiLGPUeahjyAQ73+8k geryb9ymjoDFjkKX2n7YxCHy/MnB5HayNIuUPi+KUFzpradx1v7S54XL2DHm mDRR2DDeou9H6WcIqknRh4e6fc1a70E2CbpKr9qu7AiNiEfRZzXod//joavW h0MkYC0Ug41UG64R9QTCJOKp+wSjri+IUgSSrs3WPYXb5W1jZPFIhsFkigws VgitZTv3+rO5ZyHbtCR+3yNI5isU18Lhf+Dr01MExUuyCQQz6zODXV0W+xgP wsMSe8ZXXr84a/8MKoP90mr2pNiiasMwWrcZ/klQ9J4AIqh8DJEHJeAWf+4N pYWTiRFbq3NZzIUjTBqtP/AliKvCTDQhVP3E8hK1qYg4Gv0gQ0Zu76F5c5/p rj9HTZa+o8rSQM0TDuiqKSMEJUcuMt/TScWmQNZF1GTb3HSx6LW6H+aOkLuE N0Fi+rkYupxXC3P3HnU35GMzlum//j/svIFkLOA5V5abVAttcxrGg9jpebUO i3f4DR6e86RNLMaakNoybYlK9J+7j3JjKydBTqkDn9sKBeMaE/oW21Ft99/z eJDLf+8xGt02tV512mPDw8SWJZUws3/B4qc4yrkYUe2aWBeHrE7vIX8ZgC1M icrE =/pQd -----END PGP SIGNATURE----- -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
