Le 04/07/2013 01:07, Vladislav Gorbunov a écrit :
ceph osd pool set data pg_num 1800
And I do not understand why the OSD 16 and 19 are hardly used
Actually you need to change the pgp_num for real data rebalancing:
ceph osd pool set data pgp_num 1800
Check it with the command:
ceph osd dump | grep 'pgp_num'
Vladislav Gorbunov, Michael Lowe, Thank you. I think it's that.
I changed the value and the placement is really better, but the
operation will be long ;) :
2013-07-04 18:39:03.022694 mon.0 [INF] pgmap v210884: 1928 pgs: 412
active+clean, 1115 active+remapped+wait_backfill, 303
active+remapped+wait_backfill+backfill_toofull, 1
active+degraded+wait_backfill+backfill_toofull, 86
active+remapped+backfilling, 6 active+remapped+backfill_toofull, 3
active+recovery_wait+remapped, 2
active+degraded+remapped+wait_backfill+backfill_toofull; 345 TB data,
49504 GB used, 17499 GB / 67004 GB avail; 92830230/261366383 degraded
(35.517%); recovering 94 o/s, 375MB/s
jack
67 -> 77% /var/lib/ceph/osd/ceph-6
86 -> 84% /var/lib/ceph/osd/ceph-8
77 -> 86% /var/lib/ceph/osd/ceph-11
66 -> 80% /var/lib/ceph/osd/ceph-7
47 -> 55% /var/lib/ceph/osd/ceph-10
29 -> 52% /var/lib/ceph/osd/ceph-9
joe
86 -> 86% /var/lib/ceph/osd/ceph-15
67 -> 70% /var/lib/ceph/osd/ceph-13
96 -> 88% /var/lib/ceph/osd/ceph-14
85 -> 86% /var/lib/ceph/osd/ceph-17
87 -> 84% /var/lib/ceph/osd/ceph-12
20 -> 38% /var/lib/ceph/osd/ceph-16
wiliam
86 -> 86% /var/lib/ceph/osd/ceph-0
86 -> 86% /var/lib/ceph/osd/ceph-3
61 -> 68% /var/lib/ceph/osd/ceph-4
71 -> 83% /var/lib/ceph/osd/ceph-1
58 -> 68% /var/lib/ceph/osd/ceph-18
50 -> 63% /var/lib/ceph/osd/ceph-2
Thanks a lot to Alex Bligh and Gregory Farnum
For the other question, can we change the ratio of 95% because with hard
disks 4T it makes at least 200G lost by OSD and with 18 OSD it makes 3.5To.
Regards
2013/7/3 Pierre BLONDEAU <pierre.blondeau@xxxxxxxxxx>:
Le 01/07/2013 19:17, Gregory Farnum a écrit :
On Mon, Jul 1, 2013 at 10:13 AM, Alex Bligh <alex@xxxxxxxxxxx> wrote:
On 1 Jul 2013, at 17:37, Gregory Farnum wrote:
Oh, that's out of date! PG splitting is supported in Cuttlefish:
"ceph osd pool set <foo> pg_num <number>"
http://ceph.com/docs/master/rados/operations/control/#osd-subsystem
Ah, so:
pg_num: The placement group number.
means
pg_num: The number of placement groups.
Perhaps worth demystifying for those hard of understanding such as
myself.
I'm still not quite sure how that relates to pgp_num.
Pools are sharded into placement groups. That's the pg_num. Those
placement groups can be placed all independently, or as if there were
a smaller number of placement groups (this is so you can double the
number of PGs but not move any data until the splitting is done).
-Greg
Hy,
Thank you very much for your answer. Sorry for the late reply but a
modification of a cluster of 67T is long ;)
Actually my pg number was very insufficient :
ceph osd pool get data pg_num
pg_num: 48
As I'm not sure of the rate of replication that I will set, I change the
number of pg to 1800:
ceph osd pool set data pg_num 1800
But the placement is always heterogeneous especially on the machine where I
had an full osd. I now have two osd on this machine to the limit and I can
not write to the cluster
jack
67 -> 67% /var/lib/ceph/osd/ceph-6
86 -> 86% /var/lib/ceph/osd/ceph-8
85 -> 77% /var/lib/ceph/osd/ceph-11
? -> 66% /var/lib/ceph/osd/ceph-7
47 -> 47% /var/lib/ceph/osd/ceph-10
29 -> 29% /var/lib/ceph/osd/ceph-9
joe
86 -> 77% /var/lib/ceph/osd/ceph-15
67 -> 67% /var/lib/ceph/osd/ceph-13
95 -> 96% /var/lib/ceph/osd/ceph-14
92 -> 95% /var/lib/ceph/osd/ceph-17
86 -> 87% /var/lib/ceph/osd/ceph-12
20 -> 20% /var/lib/ceph/osd/ceph-16
william
68 -> 86% /var/lib/ceph/osd/ceph-0
86 -> 86% /var/lib/ceph/osd/ceph-3
67 -> 61% /var/lib/ceph/osd/ceph-4
79 -> 71% /var/lib/ceph/osd/ceph-1
58 -> 58% /var/lib/ceph/osd/ceph-18
64 -> 50% /var/lib/ceph/osd/ceph-2
ceph -w :
2013-07-03 10:56:06.610928 mon.0 [INF] pgmap v174071: 1928 pgs: 1816
active+clean, 84 active+remapped+backfill_toofull, 9
active+degraded+backfill_toofull, 19
active+degraded+remapped+backfill_toofull; 300 TB data, 45284 GB used, 21719
GB / 67004 GB avail; 15EB/s rd, 15EB/s wr, 15Eop/s; 9975324/165229620
degraded (6.037%); recovering 15E o/s, 15EB/s
2013-07-03 10:56:08.404701 osd.14 [WRN] OSD near full (95%)
2013-07-03 10:56:29.729297 osd.17 [WRN] OSD near full (94%)
And I do not understand why the OSD 16 and 19 are hardly used
Regards
--
----------------------------------------------
Pierre BLONDEAU
Administrateur Systèmes & réseaux
Université de Caen
Laboratoire GREYC, Département d'informatique
tel : 02 31 56 75 42
bureau : Campus 2, Science 3, 406
----------------------------------------------
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
----------------------------------------------
Pierre BLONDEAU
Administrateur Systèmes & réseaux
Université de Caen
Laboratoire GREYC, Département d'informatique
tel : 02 31 56 75 42
bureau : Campus 2, Science 3, 406
----------------------------------------------
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
