|
Two weeks ago, we started a data migration from one old ceph node to a new one. For task we added a 120TB Host to the cluster and evacuated the old one with the ceph osd crush reweight osd.X 0.0 that move near 15 TB per day. After 1 week and few days we found that balancer module don't work fine under this situacion it don't distribute data between OSD if cluster is not HEALTH status. The current situation , some osd are at 96% and others at 75% , causing some pools get very nearfull 99%. I read several post about balancer only works in HEALHTY mode and that's the problem, because ceph don't distribute the data equal between OSD in native mode, causing in the scenario of "Evacuate+Add" huge problems. Info: https://pastebin.com/HuEt5Ukn Right now for solve we are manually change weight of most used osd. Anyone more got this problem?
You can determine your biggest pools like this:
``` (header="pool objects bytes_used max_avail"; echo "$header"; echo "$header" | tr '[[:alpha:]_]' '-'; ceph df --format=json | jq '.pools[]|(.name,.stats.objects,.stats.bytes_used,.stats.max_avail)' | paste - - - -) | column -t ```
Then you can select your PGs for this pool:
``` (header="pg_id pg_used_mbytes pg_used_objects" ; echo "$header" ; echo "$header" | tr '[[:alpha:]_]' '-' ; ceph pg ls-by-pool <pool_name> --format=json | jq 'sort_by(.stat_sum.num_bytes) | .[] | (.pgid, .stat_sum.num_bytes/1024/1024, .stat_sum.num_objects)' | paste - - -) | column -t ```
And then upmap your biggest PG's to lower filled osd's. Or another way, list PG's of your already nearfull osds like this `ceph pg ls-by-osd osd.0` and upmap it from this osd to lower filled osd's.
gl, k
|
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
