Is it safe to increase pg_num and pgp_num from 1024 up to 2048 for volumes and default.rgw.buckets.data pools?
How will it impact cluster behavior? I guess cluster rebalancing will occur and will take long time considering amount of data we have on it ?
Regards
Jakub
On Wed, Jan 31, 2018 at 1:37 PM, Jakub Jaszewski <jaszewski.jakub@xxxxxxxxx> wrote:
Hi,
I'm wondering why slow requests are being reported mainly when the request has been put into the queue for processing by its PG (queued_for_pg , http://docs.ceph.com/docs/master/rados/troubleshooting/ ).troubleshooting-osd/# debugging-slow-request
Could it be due too low pg_num/pgp_num ?
It looks that slow requests are mainly addressed to default.rgw.buckets.data (pool id 20) , volumes (pool id 3) and default.rgw.buckets.index (pool id 14)
2018-01-31 12:06:55.899557 osd.59 osd.59 10.212.32.22:6806/4413 38 : cluster [WRN] slow request 30.125793 seconds old, received at 2018-01-31 12:06:25.773675: osd_op(client.857003.0:126171692 3.a4fec1ad 3.a4fec1ad (undecoded) ack+ondisk+write+known_if_ redirected e5722) currently queued_for_pg
Btw how can I get more human-friendly client information from log entry like above ?
Current pg_num/pgp_numpool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool stripe_width 0 application rbdremoved_snaps [1~3]pool 14 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool stripe_width 4224 application rgw
Usage
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
385T 144T 241T 62.54 31023k
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED
volumes 3 N/A N/A 40351G 70.91 16557G 10352314 10109k 2130M 2520M 118T
default.rgw.buckets.index 14 N/A N/A 0 0 16557G 205 205 160M 27945k 0
default.rgw.buckets.data 20 N/A N/A 79190G 70.51 33115G 20865953 20376k 122M 113M 116T# ceph osd pool ls detailpool 0 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 4502 flags hashpspool stripe_width 0 application rbdpool 1 'vms' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool stripe_width 0 application rbdpool 2 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 last_change 5175 flags hashpspool stripe_width 0 application rbdremoved_snaps [1~7,14~2]pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool stripe_width 0 application rbdremoved_snaps [1~3]pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 6 'default.rgw.data.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 7 'default.rgw.gc' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 8 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 9 'default.rgw.users.uid' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 10 'default.rgw.usage' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 11 'default.rgw.users.email' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgwpool 12 'default.rgw.users.keys' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgwpool 13 'default.rgw.users.swift' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 14 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 15 'default.rgw.buckets.data.old' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 16 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 17 'default.rgw.buckets.extra' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 18 '.rgw.buckets.extra' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 4502 flags hashpspool stripe_width 0 application rgwpool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4502 flags hashpspool stripe_width 4224 application rgwpool 21 'benchmark_replicated' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 4550 flags hashpspool stripe_width 0 application rbdremoved_snaps [1~3]pool 22 'benchmark_erasure_coded' erasure size 9 min_size 7 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 last_change 4552 flags hashpspool stripe_width 24576 application rbdremoved_snaps [1~3]# ceph df detailGLOBAL:SIZE AVAIL RAW USED %RAW USED OBJECTS385T 144T 241T 62.54 31023kPOOLS:NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USEDrbd 0 N/A N/A 0 0 16557G 0 0 1 134k 0vms 1 N/A N/A 0 0 16557G 0 0 0 0 0images 2 N/A N/A 7659M 0.05 16557G 1022 1022 51247 5668 22977Mvolumes 3 N/A N/A 40351G 70.91 16557G 10352314 10109k 2130M 2520M 118T.rgw.root 4 N/A N/A 1588 0 16557G 4 4 90 4 4764default.rgw.control 5 N/A N/A 0 0 16557G 8 8 0 0 0default.rgw.data.root 6 N/A N/A 93943 0 16557G 336 336 239k 6393 275kdefault.rgw.gc 7 N/A N/A 0 0 16557G 32 32 1773M 5281k 0default.rgw.log 8 N/A N/A 0 0 16557G 185 185 22404k 14936k 0default.rgw.users.uid 9 N/A N/A 3815 0 16557G 15 15 187k 53303 11445default.rgw.usage 10 N/A N/A 0 0 16557G 7 7 278k 556k 0default.rgw.users.email 11 N/A N/A 58 0 16557G 3 3 0 3 174default.rgw.users.keys 12 N/A N/A 177 0 16557G 10 10 262 22 531default.rgw.users.swift 13 N/A N/A 40 0 16557G 3 3 0 3 120default.rgw.buckets.index 14 N/A N/A 0 0 16557G 205 205 160M 27945k 0default.rgw.buckets.data.old 15 N/A N/A 668G 3.88 16557G 180867 176k 707k 2318k 2004Gdefault.rgw.buckets.non-ec 16 N/A N/A 0 0 16557G 114 114 17960 12024 0default.rgw.buckets.extra 17 N/A N/A 0 0 16557G 0 0 0 0 0.rgw.buckets.extra 18 N/A N/A 0 0 16557G 0 0 0 0 0default.rgw.buckets.data 20 N/A N/A 79190G 70.51 33115G 20865953 20376k 122M 113M 116Tbenchmark_replicated 21 N/A N/A 1415G 7.88 16557G 363800 355k 1338k 1251k 4247Gbenchmark_erasure_coded 22 N/A N/A 11057M 0.03 33115G 2761 2761 398 5520 16586M
ThanksJakub
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
