Re: ceph cluster having blocke requests very frequently

Peter Maloney <peter.maloney@xxxxxxxxxxxxxxxxxxxx> · Fri, 18 Nov 2016 23:15:38 +0100



    On 11/18/16 18:00, Thomas Danan wrote:

    
        I
            often read that small IO write and RBD are working better
            with bigger filestore_max_sync_interval than default value.
        Default
            value is 5 sec and I saw many post saying they are using 30
            sec.
        Also
            the slow request symptom is often linked to this parameter.
        My
            journals are 10GB ( collocated with OSD storage ) and
            overall Client IO write throughput is around 500MB/s at peak
            time.
        I
            understand that the journal is flushed when it is half
            filled or when it reach the filestore_max_sync_interval
            value.
        I
            guess I can safely give it a try ? just to see the impact ?
        Not
            sure if I can change this online ?
      
    
    yes you can do it online

    
    ceph tell osd.* injectargs --filestore-max-sync-interval=30

    
    (ignore if it says unchangable, and use  ceph daemon osd.$n config
    show| grep interval  to verify it changed)

    
    I tried setting it to 5 and 120 and didn't notice any difference in
    blocked requests, only smoother IO performance. I have 16GB
    journals.

    
    BTW, my rebalance finished, and I guess the performance is a bit
    better, with load distributed a bit better, but blocked requests
    still happen if I use snapshot create + export-diff + delete
    snapshot, and make qemu clients hang the same. A 30s sleep between
    image backups prevents most, but not large exports (or snap creates
    or snap rm?). I plan to test and see if really heavy client IO can
    cause it too (without creating more objects, like in place
    modification rather than using less space like trim and using more
    space), or only other io (rbd commands, rebalancing, cow, scrub). So
    far I suspect it has to do with either poor xattr performance, or
    something to do with a large amount of splitting or otherwise
    reorganizing many pg/object directories all at once.

  
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com