Hi again,
when setup file-fsync-freq=1 (fsync for each time writing) and file-fsync-freq=0 (never fsync by sysbench), the result gets huge difference.(one is 382.94Kb/sec, another is 25.921Mb/sec).
How do you think of it? thanks.
# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 --file-fsync-freq=1 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 1 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 4309 Read, 2873 Write, 367707 Other = 374889 Total
Read 67.328Mb Written 44.891Mb Total transferred 112.22Mb (382.94Kb/sec)
23.93 Requests/sec executed
Test execution summary:
total time: 300.0782s
total number of events: 7182
total time taken by event execution: 2.3207
per-request statistics:
min: 0.01ms
avg: 0.32ms
max: 80.17ms
approx. 95 percentile: 1.48ms
Threads fairness:
events (avg/stddev): 7182.0000/0.00
execution time (avg/stddev): 2.3207/0.00
# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 --file-fsync-freq=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 298613 Read, 199075 Write, 0 Other = 497688 Total
Read 4.5565Gb Written 3.0376Gb Total transferred 7.5941Gb (25.921Mb/sec)
1658.93 Requests/sec executed
Test execution summary:
total time: 300.0049s
total number of events: 497688
total time taken by event execution: 299.7026
per-request statistics:
min: 0.00ms
avg: 0.60ms
max: 2211.13ms
approx. 95 percentile: 1.21ms
Threads fairness:
events (avg/stddev): 497688.0000/0.00
execution time (avg/stddev): 299.7026/0.00
2016-05-25 15:01 GMT+08:00 Ken Peng <ken@xxxxxxxxxx>:
Both data network and cluster network are 10Gbps.OS is Ubuntu 16.04 LTS, ceph version 10.2.0Hello,We have a cluster with 20+ hosts and 200+ OSDs, each 4T SATA disk for an OSD, no SSD cache.We run ceph as block storage service only (rbd client within VM).For testing within a VM with sysbench tool, we see that the seqwrite has a relatively good performance, it can reach 170.37Mb/sec, but random read/write always gets bad result, it can be only 474.63Kb/sec (shown as below).
Can you help give the idea why the random IO is so worse? Thanks.This is what sysbench outputs,
# sysbench --test=fileio --file-total-size=5G prepare
sysbench 0.4.12: multi-threaded system evaluation benchmark
128 files, 40960Kb each, 5120Mb total
Creating files for the test...
# sysbench --test=fileio --file-total-size=5G --file-test-mode=seqwr --init-rng=on --max-time=300 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential write (creation) test
Threads started!
Done.
Operations performed: 0 Read, 327680 Write, 128 Other = 327808 Total
Read 0b Written 5Gb Total transferred 5Gb (170.37Mb/sec)
10903.42 Requests/sec executed
Test execution summary:
total time: 30.0530s
total number of events: 327680
total time taken by event execution: 28.5936
per-request statistics:
min: 0.01ms
avg: 0.09ms
max: 192.84ms
approx. 95 percentile: 0.03ms
Threads fairness:
events (avg/stddev): 327680.0000/0.00
execution time (avg/stddev): 28.5936/0.00
# sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 40Mb each
5Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 5340 Read, 3560 Write, 11269 Other = 20169 Total
Read 83.438Mb Written 55.625Mb Total transferred 139.06Mb (474.63Kb/sec)
29.66 Requests/sec executed
Test execution summary:
total time: 300.0216s
total number of events: 8900
total time taken by event execution: 6.4774
per-request statistics:
min: 0.01ms
avg: 0.73ms
max: 90.18ms
approx. 95 percentile: 1.60ms
Threads fairness:
events (avg/stddev): 8900.0000/0.00
execution time (avg/stddev): 6.4774/0.00
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
