Re: ceph-users Digest, Vol 109, Issue 18

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Mark,

Yes, we are aware about  kv_sync_thread scaling on non-SPDK deployments. We
have plentiful AMD cores and Micron NVME just like on the RedHat benchmark
with them. I don't know if it has the same behavior with SPDK too. I need
to know how many cores per OSD are needed to saturate Micron NVME
namespace, with single or some namespaces per drive. Maybe RedHat has some
benchmarking data with the same AMD and Micron NVME just like before but
with SPDK.

Best regards.

>
> Date: Sat, 5 Feb 2022 07:17:14 -0600
> From: Mark Nelson <mnelson@xxxxxxxxxx>
> Subject:  Re: NVME Namspaces vs SPDK
> To: ceph-users@xxxxxxx
> Message-ID: <d0a3f6f1-8403-243d-e0dc-8f4f757fa8f6@xxxxxxxxxx>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> 4 NVMe drives per OSD only really makes sense if you have extra CPU to
> spare and very fast drives.  It can have higher absolute performance
> when you give OSDs unlimited CPUs, but it tends to be slower (and less
> efficient) in CPU limited scenarios in our testing (ymmv).  We've got
> some semi-recent CPU-limited results from the 2021 Q3 crimson slide deck
> here (see slides 28-31):
>
>
> https://docs.google.com/presentation/d/1eydyAFKRea8n-VniQzXKW8qkKM9GLVMJt2uDjipJjQA/edit?usp=sharing
>
> I don't think I've seen anyone specifically test multiple OSDs per drive
> with SPDK.  Figure that the write path often is bottlenecked in the the
> kv_sync_thread and/or rocksdb.  That's where going multi-OSD gives you
> more parallelism.  The read path can be a bit more of a wildcard but
> some times can also benefit from extra parallelism.  Still, it's only
> really worth it if you've got the CPU to back it up.
>
>
> Mark
>
>
> On 2/5/22 6:55 AM, Lazuardi Nasution wrote:
> > Hi,
> >
> > I have read some benchmarks which recommends of using 4 OSDs per NVME
> > drive. Until now, I'm using 4 NVME namespaces per drive for doing that
> way.
> > If I'm using SPDK, do I still need to follow 4 OSDs per NVME drive way?
> Is
> > there any benchmark related to SPDK and number of OSDs per NVME drive?
> >
> > Best regards.
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux