Hi,
This is a followup on the OSD wbthrottle slow issue casued by fdatasync discussed on the weekly performance meeting.
For a 4K random write tests (qd=2) with qem-rbd driver and 50 volumes:
* We are able to get ~3100IOPS on our setup with Firefly (0.80.5) default parameter
* We can get ~3200 IOPS on the same setup with if we set wbthrottle = 1
* if we replace fdatasync with sync_file_range, we are albe to get 4457 IOPS, 41% improvement compared with all the default parameter.
Setup configuration:
* 4 OSD nodes (E3-1275 @3.5GHz processor with 32GB memory)
* each attached 10x Seagate 3TB 7200rpm HDD, with 2x Intel 400GB SSD as Journal
* network is 10GB.
Wbthrottle tunings:
filestore_wbthrottle_xfs_ios_start_flusher=1
filestore_wbthrottle_xfs_bytes_start_flusher=1
filestore_wbthrottle_xfs_inodes_start_flusher=1
fdatasync -> sync_file_range:
diff codes_backup/WBThrottle.cc ceph-0.80.5/src/os/WBThrottle.cc
163c163
< #ifdef HAVE_FDATASYNC
---
> /*#ifdef HAVE_FDATASYNC
164a165,167
> */
> #ifdef HAVE_SYNC_FILE_RANGE
> ::sync_file_range(**wb.get<1>(), wb.get<2>().offset, wb.get<2>().len, SYNC_FILE_RANGE_WRITE);
210c213
< wbiter->second.first.add(nocache, len, 1);
---
> wbiter->second.first.add(nocache, len, 1, offset, len);
[root@a-ceph04 opt]# diff codes_backup/WBThrottle.h ceph-0.80.5/src/os/WBThrottle.h
72a73,74
> uint64_t offset;
> uint64_t len;
74c76
< void add(bool _nocache, uint64_t _size, uint64_t _ios) {
---
> void add(bool _nocache, uint64_t _size, uint64_t _ios, uint64_t _offset, uint64_t _len) {
78a81,82
> offset = _offset;
> len = _len;
Thanks
Jian
��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
