Thank you for reply,
I have created two class SSD and NvME and assigned them to crush maps.
$ ceph osd crush rule ls
replicated_rule
ssd_pool
nvme_pool
Running benchmarks on nvme is the worst performing. SSD showing much better
results compared to NvME. NvME model is Samsung_SSD_980_PRO_1TB
#### NvME pool benchmark with 3x replication
# rados -p test-nvme -t 64 -b 4096 bench 10 write
hints = 1
Maintaining 64 concurrent writes of 4096 bytes to objects of size 4096 for
up to 10 seconds or 0 objects
Object prefix: benchmark_data_os-ctrl1_1931595
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 -
0
1 64 5541 5477 21.3917 21.3945 0.0134898
0.0116529
2 64 11209 11145 21.7641 22.1406 0.00939951
0.0114506
3 64 17036 16972 22.0956 22.7617 0.00938263
0.0112938
4 64 23187 23123 22.5776 24.0273 0.00863939
0.0110473
5 64 29753 29689 23.1911 25.6484 0.00925603
0.0107662
6 64 36222 36158 23.5369 25.2695 0.0100759
0.010606
7 63 42997 42934 23.9551 26.4688 0.00902186
0.0104246
8 64 49859 49795 24.3102 26.8008 0.00884379
0.0102765
9 64 56429 56365 24.4601 25.6641 0.00989885
0.0102124
10 31 62727 62696 24.4869 24.7305 0.0115833
0.0102027
Total time run: 10.0064
Total writes made: 62727
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 24.4871
Stddev Bandwidth: 1.85423
Max bandwidth (MB/sec): 26.8008 <------------ Only 26MB/s for nvme
disk
Min bandwidth (MB/sec): 21.3945
Average IOPS: 6268
Stddev IOPS: 474.683
Max IOPS: 6861
Min IOPS: 5477
Average Latency(s): 0.0102022
Stddev Latency(s): 0.00170505
Max latency(s): 0.0365743
Min latency(s): 0.00641319
Cleaning up (deleting benchmark objects)
Removed 62727 objects
Clean up completed and total clean up time :8.23223
### SSD pool benchmark
(venv-openstack) root@os-ctrl1:~# rados -p test-ssd -t 64 -b 4096 bench 10
write
hints = 1
Maintaining 64 concurrent writes of 4096 bytes to objects of size 4096 for
up to 10 seconds or 0 objects
Object prefix: benchmark_data_os-ctrl1_1933383
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg
lat(s)
0 0 0 0 0 0 -
0
1 63 43839 43776 170.972 171 0.000991462
0.00145833
2 64 92198 92134 179.921 188.898 0.00211419
0.001387
3 64 141917 141853 184.675 194.215 0.00106326
0.00135174
4 63 193151 193088 188.534 200.137 0.00179379
0.00132423
5 63 243104 243041 189.847 195.129 0.000831263
0.00131512
6 63 291045 290982 189.413 187.27 0.00120208
0.00131807
7 64 341295 341231 190.391 196.285 0.00102127
0.00131137
8 63 393336 393273 191.999 203.289 0.000958149
0.00130041
9 63 442459 442396 191.983 191.887 0.00123453
0.00130053
Total time run: 10.0008
Total writes made: 488729
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 190.894
Stddev Bandwidth: 9.35224
Max bandwidth (MB/sec): 203.289
Min bandwidth (MB/sec): 171
Average IOPS: 48868
Stddev IOPS: 2394.17
Max IOPS: 52042
Min IOPS: 43776
Average Latency(s): 0.00130796
Stddev Latency(s): 0.000604629
Max latency(s): 0.0268462
Min latency(s): 0.000628738
Cleaning up (deleting benchmark objects)
Removed 488729 objects
Clean up completed and total clean up time :8.84114
On Wed, Aug 23, 2023 at 1:25 PM Adam King <adking@xxxxxxxxxx> wrote:
> this should be possible by specifying a "data_devices" and "db_devices"
> fields in the OSD spec file each with different filters. There's some
> examples in the docs
> https://docs.ceph.com/en/latest/cephadm/services/osd/#the-simple-case that
> show roughly how that's done, and some other sections (
> https://docs.ceph.com/en/latest/cephadm/services/osd/#filters) that go
> more in depth on the different filtering options available so you can try
> and find one that works for your disks. You can check the output of "ceph
> orch device ls --format json | jq" to see things like what cephadm
> considers the model, size etc. for the devices to be for use in the
> filtering.
>
> On Wed, Aug 23, 2023 at 1:13 PM Satish Patel <satish.txt@xxxxxxxxx> wrote:
>
>> Folks,
>>
>> I have 3 nodes with each having 1x NvME (1TB) and 3x 2.9TB SSD. Trying to
>> build ceph storage using cephadm on Ubuntu 22.04 distro.
>>
>> If I want to use NvME for Journaling (WAL/DB) for my SSD based OSDs then
>> how does cephadm handle it?
>>
>> Trying to find a document where I can tell cephadm to deploy wal/db on
>> nvme
>> so it can speed up write optimization. Do I need to create or cephadm will
>> create each partition for the number of OSD?
>>
>> Help me to understand how it works and is it worth doing?
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
