On Mon, Jun 25, 2018, 10:02 PM shadow_lin <shadow_lin@xxxxxxx> wrote:
Hi David,
I am afraid I can't run the command you provide now,because I tried to remove another osd on that host to see if it would make the data distribution even and it did.
The pg number of my pools are at power of 2.
Below is from my note before removed another osd:
pool 3 'ec_rbd_pool' erasure size 6 min_size 5 crush_rule 2 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 3248 flags hashpspool,ec_overwrites,nearfull stripe_width 16384 application rbd
pool 4 'rbd_pool' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 3248 flags hashpspool,nearfull stripe_width 0 application rbd
pg distribution of osd of all pools:
https://pasteboard.co/HrBZv3s.png
What I don't understand is why data distribution is uneven when pg distribution is even.
2018-06-26
shadow_lin
发件人：David Turner <drakonstein@xxxxxxxxx>
发送时间：2018-06-26 01:24
主题：Re: Uneven data distribution with even pg distribution after rebalancing
收件人："shadow_lin"<shadow_lin@xxxxxxx>
抄送："ceph-users"<ceph-users@xxxxxxxxxxxxxx>
I should be able to answer this question for you if you can supply the output of the following commands. It will print out all of your pool names along with how many PGs are in that pool. My guess is that you don't have a power of 2 number of PGs in your pool. Alternatively you might have multiple pools and the PGs from the various pools are just different sizes.
ceph osd lspools | tr ',' '\n' | awk '/^[0-9]/ {print $2}' | while read pool; do echo $pool: $(ceph osd pool get $pool pg_num | cut -d' ' -f2); done
ceph df
For me the output looks like this.
rbd: 64
cephfs_metadata: 64
cephfs_data: 256
rbd-ssd: 32
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
46053G 26751G 19301G 41.91
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd-replica 4 897G 11.36 7006G 263000
cephfs_metadata 6 141M 0.05 268G 11945
cephfs_data 7 10746G 43.41 14012G 2795782
rbd-replica-ssd 9 241G 47.30 268G 75061
On Sun, Jun 24, 2018 at 9:48 PM shadow_lin <shadow_lin@xxxxxxx> wrote:
Hi List,
The enviroment is:
Ceph 12.2.4
Balancer module on and in upmap mode
Failure domain is per host, 2 OSD per host
EC k=4 m=2
PG distribution is almost even before and after the rebalancing.
After marking out one of the osd,I noticed a lot of the data was moving into the other osd on the same host .
Ceph osd df result is(osd.20 and osd.21 are in the same host and osd.20 was marked out):
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
19 hdd 9.09560 1.00000 9313G 7079G 2233G 76.01 1.00 135
21 hdd 9.09560 1.00000 9313G 8123G 1190G 87.21 1.15 135
22 hdd 9.09560 1.00000 9313G 7026G 2287G 75.44 1.00 133
23 hdd 9.09560 1.00000 9313G 7026G 2286G 75.45 1.00 134
I am using RBD only so the objects should all be 4m .I don't understand why osd 21 got significant more data
with the same pg as other osds.
Is this behavior expected or I misconfiged something or some kind of bug?
Thanks
2018-06-25
shadow_lin
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
