Hello everyone,
I've been using ceph to provide storage using RBD for 60 KVM virtual
machines running on proxmox.
The ceph cluster we have is very small (2 OSDs + 1 mon per node, and a
total of 3 nodes) and we are having some performace issues, like big
latency times (apply lat:~0.5 s; commit lat: 0.001 s), which get worse
by the weekly deep-scrubs.
I wonder if doubling the numbers of OSDs would improve latency times, or
if there is any other configuration tweak recommended for such small
cluster. Also, I'm looking forward to read any experience of other users
using a similiar configuration.
Some technical info:
- Ceph version: 10.2.5
- OSDs have SSD journal (one SSD disk per 2 OSDs) and have a spindle
for backend disk.
- Using CFQ disk queue scheduler
- OSD configuration excerpt:
osd_recovery_max_active = 1
osd_recovery_op_priority = 63
osd_client_op_priority = 1
osd_mkfs_options = -f -i size=2048 -n size=64k
osd_mount_options_xfs = inode64,noatime,logbsize=256k
osd_journal_size = 20480
osd_op_threads = 12
osd_disk_threads = 1
osd_disk_thread_ioprio_class = idle
osd_disk_thread_ioprio_priority = 7
osd_scrub_begin_hour = 3
osd_scrub_end_hour = 8
osd_scrub_during_recovery = false
filestore_merge_threshold = 40
filestore_split_multiple = 8
filestore_xattr_use_omap = true
filestore_queue_max_ops = 2500
filestore_min_sync_interval = 0.01
filestore_max_sync_interval = 0.1
filestore_journal_writeahead = true
Best regards,
--
Fernando Cid O.
Ingeniero de Operaciones
AltaVoz S.A.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
