On 2/24/19 4:34 PM, David Turner wrote: > One thing that's worked for me to get more out of nvmes with Ceph is to > create multiple partitions on the nvme with an osd on each partition. > That way you get more osd processes and CPU per nvme device. I've heard > of people using up to 4 partitions like this. > Increasing the amount of Placement Groups also works. In addition you should also increase osd_op_num_threads_per_shard to something like 4. This will increase CPU usage, but you should also be able to get more out of the NVMe devices. In addition, make sure you pin the CPU C-States to 1 and disable powersaving for the CPU. Wido > On Sun, Feb 24, 2019, 10:25 AM Vitaliy Filippov <vitalif@xxxxxxxxxx > <mailto:vitalif@xxxxxxxxxx>> wrote: > > > We can get 513558 IOPS in 4K read per nvme by fio but only 45146 IOPS > > per OSD.by rados. > > Don't expect Ceph to fully utilize NVMe's, it's software and it's > slow :) > some colleagues tell that SPDK works out of the box, but almost > doesn't > increase performance, because the userland-kernel interaction isn't > the > bottleneck currently, it's Ceph code itself. I also tried once, but I > couldn't make it work. When I have some spare NVMe's I'll make another > attempt. > > So... try it and share your results here :) we're all interested. > > -- > With best regards, > Vitaliy Filippov > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |