Hi Steve, I'm also looking for improvements of single-thread-reads. A little bit higher values (twice?) should be possible with your config. I have 5 nodes with 60 4-TB hdds and got following: rados -p test bench -b 4194304 60 seq -t 1 --no-cleanup Total time run: 60.066934 Total reads made: 863 Read size: 4194304 Bandwidth (MB/sec): 57.469 Average Latency: 0.0695964 Max latency: 0.434677 Min latency: 0.016444 In my case I had some osds (xfs) with an high fragmentation (20%). Changing the mount options and defragmentation help slightly. Performance changes: [client] rbd cache = true rbd cache writethrough until flush = true [osd] osd mount options xfs = "rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M" osd_op_threads = 4 osd_disk_threads = 4 But I expect much more speed for an single thread... Udo On 23.07.2014 22:13, Steve Anthony wrote: > Ah, ok. That makes sense. With one concurrent operation I see numbers > more in line with the read speeds I'm seeing from the filesystems on the > rbd images. > > # rados -p bench bench 300 seq --no-cleanup -t 1 > Total time run: 300.114589 > Total reads made: 2795 > Read size: 4194304 > Bandwidth (MB/sec): 37.252 > > Average Latency: 0.10737 > Max latency: 0.968115 > Min latency: 0.039754 > > # rados -p bench bench 300 rand --no-cleanup -t 1 > Total time run: 300.164208 > Total reads made: 2996 > Read size: 4194304 > Bandwidth (MB/sec): 39.925 > > Average Latency: 0.100183 > Max latency: 1.04772 > Min latency: 0.039584 > > I really wish I could find my data on read speeds from a couple weeks > ago. It's possible that they've always been in this range, but I > remember one of my test users saturating his 1GbE link over NFS reading > copying from the rbd client to his workstation. Of course, it's also > possible that the data set he was using was cached in RAM when he was > testing, masking the lower rbd speeds. > > It just seems counterintuitive to me that read speeds would be so much > slower that writes at the filesystem layer in practice. With images in > the 10-100TB range, reading data at 20-60MB/s isn't going to be > pleasant. Can you suggest any tunables or other approaches to > investigate to improve these speeds, or are they in line with what you'd > expect? Thanks for your help! > > -Steve > >
 |