Re: EC Pool Disk Performance Toshiba vs Segate

Maged Mokhtar <mmokhtar@xxxxxxxxxxx> · Thu, 13 Dec 2018 11:18:42 +0200



    On 13/12/2018 09:53, Ashley Merrick
      wrote:

    
      I have a Mimic Bluestore EC RBD Pool running on
        8+2, this is currently running across 4 node's.
        

        3 Node's are running Toshiba disk's while one node is
          running Segate disks (same size, spinning speed, enterprise
          disks e.t.c), I have noticed huge difference in IOWAIT and
          disk latency performance between the two sets of disks, can
          also be seen from a ceph osd perf during read and write
          operations.
        

        Speaking to my host (server provider), they bench marked
          the two disks before approving them for use in this type of
          server, they actually saw higher performance from the Toshiba
          disk during their tests.
        

        They did however state there test where at higher / larger
          block sizes, I can imagine CEPH using EC of 8+2 the block
          sizes / requests are quite small?
        

        Is there anything I can do ? Changing the RBD object size
          & stripe unit to a bigger than default? Will this make the
          data sent to the disk larger chunks at once compared to lot's
          of smaller block's.
        

        If anyone else has any advice I'm open to trying.
        

        P.s I have already disabled the disk cache on all disks and
          this was causing high write latency across all.
        

        Thanks
      
      
      _______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
    
    Since you say there is a huge difference between the disk types
      under your current workload, then i would focus on this, the
      logical thing to do is to replace them. You can probably run
      further benchmarks with fsync write speed at lower block sizes,
      but i think your current observation is conclusive enough.
     
    Other less recommended options: use a lower ec profile such as k4
      m2, getting a controller with write back cache. For sequential io
      increasing your read_ahead_kb, using librbd client cache,
      adjusting your client os cache parameters. Also if you have a
      controlled application like a backup app where you can specify the
      block size, then increase it to above 1MB. But again i would
      recommend you focus on changing disks.

    
    /Maged

    
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com