Hi, I located and applied almost every different tuning setting/config over the internet. I couldn’t manage to speed up my speed one byte further. It is always same speed whatever I do. I was on jewel, now I tried BlueStore on Luminous. Still exact same speed I gain from cephfs. It doesn’t matter if I disable debug log, or remove [osd] section as below and re-add as below (see .conf). Results are exactly the same. Not a single byte is gained from those tunings. I also did tuning for kernel (sysctl.conf). Basics: I have 2 nodes with 10 OSD each and each OSD is 3TB SATA drive. Each node has 24 cores and 64GB of RAM. Ceph nodes are connected via 10GbE NIC. No FUSE used. But tried that too. Same results. $ dd if=/dev/zero of=/mnt/c/testfile bs=100M count=10 oflag=direct 10+0 records in 10+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 5.77219 s, 182 MB/s 182MB/s. This is the best speed i get so far. Usually 170~MB/s. Hm.. I get much much much higher speeds on different filesystems. Even with glusterfs. Is there anything I can do or try? Read speed is also around 180-220MB/s but not higher. This is What I am using on ceph.conf: [global] fsid = d7163667-f8c5-466b-88df-8747b26c91df mon_initial_members = server1 mon_host = 192.168.0.1 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd mount options = rw,noexec,nodev,noatime,nodiratime,nobarrier osd mount options xfs = rw,noexec,nodev,noatime,nodiratime,nobarrier osd_mkfs_type = xfs osd pool default size = 2 enable experimental unrecoverable data corrupting features = bluestore rocksdb bluestore fsck on mount = true rbd readahead disable after bytes = 0 rbd readahead max bytes = 4194304 log to syslog = false debug_lockdep = 0/0 debug_context = 0/0 debug_crush = 0/0 debug_buffer = 0/0 debug_timer = 0/0 debug_filer = 0/0 debug_objecter = 0/0 debug_rados = 0/0 debug_rbd = 0/0 debug_journaler = 0/0 debug_objectcatcher = 0/0 debug_client = 0/0 debug_osd = 0/0 debug_optracker = 0/0 debug_objclass = 0/0 debug_filestore = 0/0 debug_journal = 0/0 debug_ms = 0/0 debug_monc = 0/0 debug_tp = 0/0 debug_auth = 0/0 debug_finisher = 0/0 debug_heartbeatmap = 0/0 debug_perfcounter = 0/0 debug_asok = 0/0 debug_throttle = 0/0 debug_mon = 0/0 debug_paxos = 0/0 debug_rgw = 0/0 [osd] osd max write size = 512 osd client message size cap = 2147483648 osd mount options xfs = rw,noexec,nodev,noatime,nodiratime,nobarrier filestore xattr use omap = true osd_op_threads = 8 osd disk threads = 4 osd map cache size = 1024 filestore_queue_max_ops = 25000 filestore_queue_max_bytes = 10485760 filestore_queue_committing_max_ops = 5000 filestore_queue_committing_max_bytes = 10485760000 journal_max_write_entries = 1000 journal_queue_max_ops = 3000 journal_max_write_bytes = 1048576000 journal_queue_max_bytes = 1048576000 filestore_max_sync_interval = 15 filestore_merge_threshold = 20 filestore_split_multiple = 2 osd_enable_op_tracker = false filestore_wbthrottle_enable = false osd_client_message_size_cap = 0 osd_client_message_cap = 0 filestore_fd_cache_size = 64 filestore_fd_cache_shards = 32 filestore_op_threads = 12 As I stated above, it doesn’t matter if I have this [osd] section or not. Results are same. I am open to all suggestions. Thanks, Gencer. |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
