Hi Thomas & Anthony,
Anthony provided great recommendations.
ssd read performance:
I find the total number of pg per ssd osd too low; it can be twice as high.
hdd read performance
What makes me a little suspicious is that the maximum throughput of about
120 MB/s is exactly the maximum of a 1 Gbit/s connection.
(I have seen this in the past if the routing is not correct and if you use
VMs for testing the network could be limited.)
Can you run your performance test with higher num_jobs, e.g. 16 and post
the results.
Regards, Joachim
joachim.kraftmayer@xxxxxxxxx
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Fr., 14. März 2025 um 01:53 Uhr schrieb Anthony D'Atri <
anthony.datri@xxxxxxxxx>:
>
> > Hi,
> >
> > we are having massive performance issues with our Ceph cluster, and by
> > now I have no idea how and where to debug further.
> >
> > See attachments
>
> I see inline text. If you attached files, those don’t propagate through
> the list.
>
> > for full benchmark logs, relevant excerpts:
> >
> > HDD pool:
> >> read: IOPS=7591, BW=119MiB/s (124MB/s)(34.9GiB/300892msec)
> >
> > SSD pool:
> >> read: IOPS=2007, BW=31.4MiB/s (32.9MB/s)(9422MiB/300334msec)
> >
> > Yes, the SSD pool is in fact slower in both IOPS and data rate than the
> > HDD pool.
>
> Making some assumptions based on the below:
>
> * Twice as many HDD OSDs as SSD OSDs, so twice the ostensible potential
> for parallelism
> * The 5210 ION is QLC from 6 years ago. Its performance is significantly
> lower than your Samsungs and especially the 5300s
> * Your pg_num values are probably fine.
> * Looks like you are benching against a KRBD mount, so I can’t tell which
> pool that’s against, or which media your pools are using. Note that your
> EC RBD pool is going to be inherently slow with high latency.
> * An RBD pool will be limited by its slowest OSDs. Half of your SSDs are
> 5210s. So … oh man this is triggering my combinatorics PTSD … many of your
> PGs in RBD pools using the SSDs will include at least one of them. The
> proportion depends on which systems they’re in, which we can’t readily tell
> from the supplied data.
> * being respectful of your resources, your systems as described are pretty
> light CPU-wise for their duties, especially if those collocated VMs are
> using much at all.
> * The OSDs on Pileus especially are light on available RAM. And those
> include some of the SSDs.
> * The RAM usage of VMs on the other systems may similarly be starving
> their OSDs.
>
> If you could post `ceph osd metadata` somewhere and send the list a link
> we can be more specific.
>
> Same with `ceph osd crush dump`
> —aad
>
>
>
> > Of course this is only one particular benchmark scenario, but
> > at least it has some numbers and not just “everything feels slow”. I’m
> > happy to run different benchmarks if required.
> >
> > Cluster information:
> > * 2x10G ethernet to a Cisco Catalyst 3850 on each node
> > * Debian 11 (bullseye), Linux 5.10
> > * Ceph 18.2.4 (reef), managed with cephadm
> > * All but the oldest node serve as VM hypervisors (KVM) as well
> > * 6 nodes (hostnames appended to compare with OSD information later)
> > - 2x AMD EPYC 7232P, 64G RAM (cirrus, nimbus)
> > - 1x Intel Xeon E3-1230 v3, 32G RAM (pileus, no VMs)
> > - 2x Intel Xeon E5-1620 v4, 64G RAM (lenticular, nacreous)
> > - 1x Intel Xeon Silver 4110, 96G RAM (stratus)
> > * Cluster usage: mostly RBD, some RGW and CephFS
> > * OSDs:
> > - 19 Hitachi Ultrastar 7K4000 2TB
> > - 1 WD Ultrastar DC HC310 4TB
> > - 3 Hitachi Deskstar 7k2000 2TB
> > - 13 Hitachi Ultrastar A7K3000 2TB
> > - 8 Micron 5210 ION 2TB
> > - 4 Micron 5300 MAX - Mixed Use 2TB
> > - 5 Samsung Datacenter PM893 2TB
> >
> > You can find various information that I deemed useful below. Please ask
> > if you would like further information.
> >
> > As I said, I don’t really know where to look and what to do, so I would
> > really appreciate any pointers on how to debug this and improve
> > performance.
> >
> > Of course I have already seen the pg warnings, but I am really not sure
> > what to adjust in which direction, especially the contradiction between
> > the MANY_OBJECTS_PER_PG and POOL_TOO_MANY_PGS warnings for the
> > default.rgw.ec.data pool (there are too many objects per pg, so I
> > should probably increase pg_num, however it recommends to scale down
> > from 128 to 32?!).
> >
> > Thanks,
> > Thomas
> >
> > # ceph -s
> > cluster:
> > id: 91688ac0-1b4f-43e3-913d-a844338d9325
> > health: HEALTH_WARN
> > 1 pools have many more objects per pg than average
> > 3 pools have too many placement groups
> >
> > services:
> > mon: 5 daemons, quorum nimbus,lenticular,stratus,cirrus,nacreous (age
> 5d)
> > mgr: lenticular(active, since 3M), standbys: nimbus, stratus,
> nacreous, cirrus
> > mds: 1/1 daemons up, 1 standby
> > osd: 53 osds: 53 up (since 2w), 53 in (since 18M)
> > rgw: 5 daemons active (5 hosts, 1 zones)
> >
> > data:
> > volumes: 1/1 healthy
> > pools: 15 pools, 2977 pgs
> > objects: 7.20M objects, 15 TiB
> > usage: 42 TiB used, 55 TiB / 97 TiB avail
> > pgs: 2975 active+clean
> > 1 active+clean+scrubbing
> > 1 active+clean+scrubbing+deep
> >
> > io:
> > client: 6.5 MiB/s rd, 1.3 MiB/s wr, 158 op/s rd, 63 op/s wr
> >
> > # ceph health detail
> > HEALTH_WARN 1 pools have many more objects per pg than average; 3 pools
> have too many placement groups
> > [WRN] MANY_OBJECTS_PER_PG: 1 pools have many more objects per pg than
> average
> > pool default.rgw.ec.data objects per pg (27197) is more than 11.2524
> times cluster average (2417)
> > [WRN] POOL_TOO_MANY_PGS: 3 pools have too many placement groups
> > Pool default.rgw.ec.data has 128 placement groups, should have 32
> > Pool templates has 512 placement groups, should have 256
> > Pool rbd.ec has 1024 placement groups, should have 256
> >
> > # ceph df
> > --- RAW STORAGE ---
> > CLASS SIZE AVAIL USED RAW USED %RAW USED
> > hdd 67 TiB 29 TiB 39 TiB 39 TiB 57.58
> > ssd 30 TiB 26 TiB 3.7 TiB 3.7 TiB 12.37
> > TOTAL 97 TiB 55 TiB 42 TiB 42 TiB 43.74
> >
> > --- POOLS ---
> > POOL ID PGS STORED OBJECTS USED %USED
> MAX AVAIL
> > templates 1 512 4.4 TiB 1.27M 13 TiB 37.46
> 7.4 TiB
> > .rgw.root 9 32 1.4 KiB 4 768 KiB 0
> 7.4 TiB
> > default.rgw.control 10 32 0 B 8 0 B 0
> 7.4 TiB
> > default.rgw.meta 11 32 7.0 KiB 38 5.1 MiB 0
> 7.4 TiB
> > default.rgw.log 12 32 3.5 KiB 208 5.7 MiB 0
> 7.4 TiB
> > default.rgw.ec.data 14 128 1.3 TiB 3.48M 2.7 TiB 10.95
> 13 TiB
> > default.rgw.buckets.index 15 32 57 MiB 53 171 MiB 0
> 7.4 TiB
> > default.rgw.buckets.data 16 32 2.5 GiB 17.85k 8.3 GiB 0.04
> 7.4 TiB
> > default.rgw.buckets.non-ec 17 32 1.6 KiB 1 197 KiB 0
> 7.4 TiB
> > rbd.ssd 19 512 872 GiB 259.98k 2.6 TiB 9.46
> 8.1 TiB
> > .mgr 20 1 640 MiB 161 1.9 GiB 0
> 7.4 TiB
> > cephfs_metadata 21 32 404 MiB 35.40k 1.2 GiB 0
> 8.1 TiB
> > cephfs_data 22 32 42 GiB 28.09k 128 GiB 0.56
> 7.4 TiB
> > rbd 23 512 12 GiB 6.57k 39 GiB 0.17
> 7.4 TiB
> > rbd.ec 24 1024 8 TiB 2.10M 16 TiB 42.48
> 11 TiB
> >
> > The pool "templates" is named as such for historical reasons. It is
> > used as RBD VM image storage.
> >
> > # ceph osd pool autoscale-status
> > POOL SIZE TARGET SIZE RATE RAW
> CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM
> AUTOSCALE BULK
> > .rgw.root 256.0k 3.0
> 68931G 0.0000 1.0 32
> off False
> > default.rgw.control 0 3.0
> 68931G 0.0000 1.0 32
> off False
> > default.rgw.meta 1740k 3.0
> 68931G 0.0000 1.0 32
> off False
> > default.rgw.log 1932k 3.0
> 68931G 0.0000 1.0 32
> off False
> > default.rgw.ec.data 1678G 1.6666666269302368
> 68931G 0.0406 1.0 128
> warn False
> > default.rgw.buckets.index 58292k 3.0
> 68931G 0.0000 1.0 32
> off False
> > default.rgw.buckets.data 2828M 3.0
> 68931G 0.0001 1.0 32
> off False
> > default.rgw.buckets.non-ec 67159 3.0
> 68931G 0.0000 1.0 32
> off False
> > .mgr 639.7M 3.0
> 68931G 0.0000 1.0 1
> off False
> > cephfs_metadata 405.4M 3.0
> 30404G 0.0000 4.0 32
> off False
> > cephfs_data 43733M 3.0
> 68931G 0.0019 1.0 32
> off False
> > templates 4541G 3.0
> 68931G 0.1977 1.0 512
> warn True
> > rbd.ssd 870.4G 3.0
> 30404G 0.0859 1.0 512
> warn True
> > rbd 13336M 3.0
> 68931G 0.0006 1.0 512
> off True
> > rbd.ec 8401G 2.0
> 68931G 0.2438 1.0 1024
> warn True
> >
> > # ceph osd pool ls detail
> > pool 1 'templates' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 7495052
> lfor 0/2976944/7469772 flags hashpspool,selfmanaged_snaps,bulk stripe_width
> 0 application rbd read_balance_score 1.48
> > pool 9 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 7467477 flags
> hashpspool stripe_width 0 application rgw read_balance_score 3.38
> > pool 10 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change
> 7467478 flags hashpspool stripe_width 0 application rgw read_balance_score
> 3.37
> > pool 11 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change
> 7467479 flags hashpspool stripe_width 0 application rgw read_balance_score
> 3.36
> > pool 12 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change
> 7467480 flags hashpspool stripe_width 0 application rgw read_balance_score
> 4.52
> > pool 14 'default.rgw.ec.data' erasure profile rgw-video size 5 min_size
> 4 crush_rule 3 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode
> warn last_change 7494999 lfor 0/5467738/7467533 flags hashpspool
> stripe_width 12288 application rgw
> > pool 15 'default.rgw.buckets.index' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off
> last_change 7467482 flags hashpspool stripe_width 0 application rgw
> read_balance_score 4.52
> > pool 16 'default.rgw.buckets.data' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off
> last_change 7467483 flags hashpspool stripe_width 0 application rgw
> read_balance_score 3.39
> > pool 17 'default.rgw.buckets.non-ec' replicated size 3 min_size 2
> crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off
> last_change 7467484 flags hashpspool stripe_width 0 application rgw
> read_balance_score 3.38
> > pool 19 'rbd.ssd' replicated size 3 min_size 2 crush_rule 4 object_hash
> rjenkins pg_num 512 pgp_num 512 autoscale_mode warn last_change 7495053
> lfor 0/141997/7467030 flags hashpspool,selfmanaged_snaps,bulk stripe_width
> 0 application rbd read_balance_score 1.43
> > pool 20 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 7480977 flags
> hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth
> read_balance_score 37.50
> > pool 21 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 4
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change
> 7467487 lfor 0/0/96365 flags hashpspool stripe_width 0 pg_autoscale_bias 4
> pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.59
> > pool 22 'cephfs_data' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change
> 7467488 lfor 0/0/96365 flags hashpspool,selfmanaged_snaps stripe_width 0
> application cephfs read_balance_score 4.52
> > pool 23 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 512 pgp_num 512 autoscale_mode off last_change 7467489 lfor
> 0/0/884114 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0
> application rbd read_balance_score 1.41
> > pool 24 'rbd.ec' erasure profile rbd size 4 min_size 3 crush_rule 5
> object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn
> last_change 7495054 lfor 0/2366959/2366961 flags
> hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 8192
> application rbd
> >
> > # ceph osd utilization
> > avg 192.66
> > stddev 75.9851 (expected baseline 13.7486)
> > min osd.48 with 88 pgs (0.456762 * mean)
> > max osd.10 with 500 pgs (2.59524 * mean)
> >
> > # ceph osd df tree
> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
> META AVAIL %USE VAR PGS STATUS TYPE NAME
> > -13 95.71759 - 97 TiB 42 TiB 42 TiB 4.2 GiB 204
> GiB 55 TiB 43.74 1.00 - root default
> > -16 14.15759 - 15 TiB 8.2 TiB 8.2 TiB 713 MiB 29
> GiB 6.4 TiB 56.36 1.29 - host cirrus
> > 2 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.0 TiB 85 MiB 4.9
> GiB 784 GiB 57.94 1.32 223 up osd.2
> > 5 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 50 MiB 3.2
> GiB 821 GiB 55.95 1.28 223 up osd.5
> > 8 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 88 MiB 2.9
> GiB 804 GiB 56.86 1.30 229 up osd.8
> > 11 hdd 1.76970 1.00000 1.8 TiB 963 GiB 959 GiB 236 MiB 3.6
> GiB 900 GiB 51.66 1.18 240 up osd.11
> > 13 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 61 MiB 4.7
> GiB 823 GiB 55.81 1.28 220 up osd.13
> > 15 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 52 MiB 3.1
> GiB 762 GiB 59.13 1.35 229 up osd.15
> > 18 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.0 TiB 85 MiB 2.7
> GiB 787 GiB 57.76 1.32 228 up osd.18
> > 19 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 54 MiB 3.4
> GiB 824 GiB 55.79 1.28 219 up osd.19
> > -18 15.93417 - 16 TiB 6.0 TiB 6.0 TiB 663 MiB 23
> GiB 10 TiB 37.15 0.85 - host lenticular
> > 0 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 88 MiB 3.2
> GiB 796 GiB 57.26 1.31 229 up osd.0
> > 4 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 42 MiB 6.0
> GiB 753 GiB 59.57 1.36 231 up osd.4
> > 6 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 3.5
> GiB 758 GiB 59.31 1.36 237 up osd.6
> > 10 hdd 3.63869 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 127 MiB 5.9
> GiB 1.7 TiB 52.08 1.19 500 up osd.10
> > 36 ssd 1.74660 1.00000 1.7 TiB 229 GiB 228 GiB 92 MiB 959
> MiB 1.5 TiB 12.79 0.29 103 up osd.36
> > 37 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 97 MiB 1006
> MiB 1.5 TiB 12.49 0.29 98 up osd.37
> > 38 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 61 MiB 1.0
> GiB 1.5 TiB 12.46 0.28 98 up osd.38
> > 39 ssd 1.74660 1.00000 1.7 TiB 221 GiB 219 GiB 102 MiB 1.4
> GiB 1.5 TiB 12.35 0.28 98 up osd.39
> > -17 22.89058 - 23 TiB 9.5 TiB 9.4 TiB 1.1 GiB 45
> GiB 14 TiB 40.59 0.93 - host nacreous
> > 1 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 124 MiB 6.0
> GiB 818 GiB 56.07 1.28 229 up osd.1
> > 3 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 3.1
> GiB 780 GiB 58.11 1.33 222 up osd.3
> > 7 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 48 MiB 3.1
> GiB 773 GiB 58.48 1.34 237 up osd.7
> > 9 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 50 MiB 4.9
> GiB 766 GiB 58.88 1.35 223 up osd.9
> > 14 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 93 MiB 3.3
> GiB 812 GiB 56.43 1.29 224 up osd.14
> > 16 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 43 MiB 5.0
> GiB 792 GiB 57.51 1.31 235 up osd.16
> > 17 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 68 MiB 4.2
> GiB 777 GiB 58.31 1.33 227 up osd.17
> > 20 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 66 MiB 5.9
> GiB 780 GiB 58.15 1.33 231 up osd.20
> > 48 ssd 1.74660 1.00000 1.7 TiB 213 GiB 212 GiB 94 MiB 1.3
> GiB 1.5 TiB 11.92 0.27 88 up osd.48
> > 49 ssd 1.74660 1.00000 1.7 TiB 215 GiB 212 GiB 94 MiB 2.5
> GiB 1.5 TiB 12.01 0.27 90 up osd.49
> > 50 ssd 1.74660 1.00000 1.7 TiB 214 GiB 212 GiB 95 MiB 2.1
> GiB 1.5 TiB 11.99 0.27 92 up osd.50
> > 51 ssd 1.74660 1.00000 1.7 TiB 217 GiB 215 GiB 144 MiB 1.9
> GiB 1.5 TiB 12.13 0.28 94 up osd.51
> > 52 ssd 1.74660 1.00000 1.7 TiB 214 GiB 212 GiB 144 MiB 2.0
> GiB 1.5 TiB 11.95 0.27 94 up osd.52
> > -19 6.98639 - 7.0 TiB 895 GiB 889 GiB 345 MiB 5.6
> GiB 6.1 TiB 12.51 0.29 - host nimbus
> > 44 ssd 1.74660 1.00000 1.7 TiB 227 GiB 224 GiB 148 MiB 2.3
> GiB 1.5 TiB 12.67 0.29 98 up osd.44
> > 45 ssd 1.74660 1.00000 1.7 TiB 222 GiB 221 GiB 68 MiB 1021
> MiB 1.5 TiB 12.39 0.28 93 up osd.45
> > 46 ssd 1.74660 1.00000 1.7 TiB 223 GiB 222 GiB 64 MiB 972
> MiB 1.5 TiB 12.48 0.29 97 up osd.46
> > 47 ssd 1.74660 1.00000 1.7 TiB 224 GiB 222 GiB 64 MiB 1.3
> GiB 1.5 TiB 12.52 0.29 95 up osd.47
> > -15 21.19368 - 22 TiB 9.3 TiB 9.3 TiB 848 MiB 56
> GiB 12 TiB 43.35 0.99 - host pileus
> > 12 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 54 MiB 8.7
> GiB 790 GiB 57.61 1.32 231 up osd.12
> > 21 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 46 MiB 6.1
> GiB 765 GiB 58.94 1.35 236 up osd.21
> > 22 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 60 MiB 6.2
> GiB 810 GiB 56.52 1.29 228 up osd.22
> > 23 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 83 MiB 8.0
> GiB 779 GiB 58.16 1.33 230 up osd.23
> > 24 hdd 1.76970 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 44 MiB 5.5
> GiB 777 GiB 58.29 1.33 229 up osd.24
> > 25 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 73 MiB 5.6
> GiB 799 GiB 57.09 1.31 230 up osd.25
> > 26 hdd 1.76970 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 63 MiB 5.4
> GiB 797 GiB 57.22 1.31 239 up osd.26
> > 27 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 99 MiB 7.1
> GiB 720 GiB 61.36 1.40 247 up osd.27
> > 40 ssd 1.74660 1.00000 1.7 TiB 220 GiB 219 GiB 66 MiB 1013
> MiB 1.5 TiB 12.31 0.28 96 up osd.40
> > 41 ssd 1.74660 1.00000 1.7 TiB 221 GiB 220 GiB 63 MiB 981
> MiB 1.5 TiB 12.36 0.28 97 up osd.41
> > 42 ssd 1.74660 1.00000 1.7 TiB 228 GiB 227 GiB 98 MiB 999
> MiB 1.5 TiB 12.73 0.29 101 up osd.42
> > 43 ssd 1.74660 1.00000 1.7 TiB 227 GiB 226 GiB 99 MiB 866
> MiB 1.5 TiB 12.67 0.29 100 up osd.43
> > -14 14.55518 - 15 TiB 8.6 TiB 8.5 TiB 582 MiB 45
> GiB 6.0 TiB 59.02 1.35 - host stratus
> > 28 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 54 MiB 5.5
> GiB 756 GiB 59.41 1.36 239 up osd.28
> > 29 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 44 MiB 4.7
> GiB 759 GiB 59.27 1.35 230 up osd.29
> > 30 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 68 MiB 5.0
> GiB 753 GiB 59.57 1.36 232 up osd.30
> > 31 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 64 MiB 6.2
> GiB 778 GiB 58.26 1.33 229 up osd.31
> > 32 hdd 1.81940 1.00000 1.8 TiB 1.0 TiB 1.0 TiB 85 MiB 5.0
> GiB 797 GiB 57.24 1.31 226 up osd.32
> > 33 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 112 MiB 4.1
> GiB 749 GiB 59.81 1.37 235 up osd.33
> > 34 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 46 MiB 8.5
> GiB 770 GiB 58.66 1.34 238 up osd.34
> > 35 hdd 1.81940 1.00000 1.8 TiB 1.1 TiB 1.1 TiB 109 MiB 6.2
> GiB 747 GiB 59.92 1.37 244 up osd.35
> > TOTAL 97 TiB 42 TiB 42 TiB 4.2 GiB 204
> GiB 55 TiB 43.74
> > MIN/MAX VAR: 0.27/1.40 STDDEV: 21.24
> >
> > # ceph balancer status
> > {
> > "active": true,
> > "last_optimize_duration": "0:00:00.040070",
> > "last_optimize_started": "Thu Mar 13 14:59:05 2025",
> > "mode": "upmap",
> > "no_optimization_needed": true,
> > "optimize_result": "Unable to find further optimization, or pool(s)
> pg_num is decreasing, or distribution is already perfect",
> > "plans": []
> > }
> >
> > --
> > Fachschaft I/1 Mathematik/Physik/Informatik der RWTH Aachen Thomas
> > Schneider Campus Mitte: Augustinerbach 2a, 52062 Aachen Telefon: +49
> > 241 80 94506 Informatikzentrum: Ahornstraße 55, Raum 4U17, 52074 Aachen
> > Telefon: +49 241 80 26741 https://www.fsmpi.rwth-aachen.de
> > # fio --ioengine=libaio --direct=1 --bs=16384 --iodepth=128
> --rw=randread --norandommap --size=20G --numjobs=1 --runtime=300
> --time_based --name=/dev/rbd/rbd/test
> > /dev/rbd/rbd/test: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W)
> 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128
> > fio-3.25
> > Starting 1 process
> > Jobs: 1 (f=1): [r(1)][100.0%][r=35.0MiB/s][r=2241 IOPS][eta 00m:00s]
> > /dev/rbd/rbd/test: (groupid=0, jobs=1): err= 0: pid=3832471: Thu Jan 30
> 19:55:46 2025
> > read: IOPS=7591, BW=119MiB/s (124MB/s)(34.9GiB/300892msec)
> > slat (usec): min=2, max=665, avg= 8.93, stdev= 5.57
> > clat (usec): min=65, max=5330.5k, avg=16850.13, stdev=184056.88
> > lat (usec): min=82, max=5330.5k, avg=16859.23, stdev=184056.97
> > clat percentiles (usec):
> > | 1.00th=[ 147], 5.00th=[ 180], 10.00th=[ 202],
> > | 20.00th=[ 237], 30.00th=[ 269], 40.00th=[ 297],
> > | 50.00th=[ 330], 60.00th=[ 367], 70.00th=[ 420],
> > | 80.00th=[ 523], 90.00th=[ 1876], 95.00th=[ 6128],
> > | 99.00th=[ 362808], 99.50th=[1098908], 99.90th=[3506439],
> > | 99.95th=[4143973], 99.99th=[4731175]
> > bw ( KiB/s): min= 768, max=478432, per=100.00%, avg=121819.01,
> stdev=95962.09, samples=600
> > iops : min= 48, max=29902, avg=7613.70, stdev=5997.62,
> samples=600
> > lat (usec) : 100=0.01%, 250=24.52%, 500=53.85%, 750=8.61%, 1000=1.56%
> > lat (msec) : 2=1.84%, 4=3.46%, 10=2.34%, 20=1.12%, 50=0.91%
> > lat (msec) : 100=0.40%, 250=0.31%, 500=0.13%, 750=0.18%, 1000=0.21%
> > lat (msec) : 2000=0.34%, >=2000=0.20%
> > cpu : usr=2.33%, sys=10.06%, ctx=1687235, majf=0, minf=523
> > IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,
> >=64=100.0%
> > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.0%
> > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
> >=64=0.1%
> > issued rwts: total=2284212,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> > latency : target=0, window=0, percentile=100.00%, depth=128
> >
> > Run status group 0 (all jobs):
> > READ: bw=119MiB/s (124MB/s), 119MiB/s-119MiB/s (124MB/s-124MB/s),
> io=34.9GiB (37.4GB), run=300892-300892msec
> > <fio-ssd.txt>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
