Re: RBD caching on 4K reads???

Udo Lembke <ulembke@xxxxxxxxxxxx> · Fri, 30 Jan 2015 22:09:32 +0100



    Hi Bruce,

    you can also look on the mon, like

    ceph --admin-daemon /var/run/ceph/ceph-mon.b.asok config show | grep
    cache

    
    (I guess you have an number instead of the .b. )

    
    Udo

    On 30.01.2015 22:02, Bruce McFarland
      wrote:

    
        The
            ceph daemon isn’t running on the client with the rbd device
            so I can’t verify if it’s disabled at the librbd level on
            the client. If you mean on the storage nodes I’ve had some
            issues dumping the config. Does the rbd caching occur on the
            storage nodes, client, or both?
         
         
            From:
                Udo Lembke [mailto:ulembke@xxxxxxxxxxxx]
                

                Sent: Friday, January 30, 2015 1:00 PM

                To: Bruce McFarland; ceph-users@xxxxxxxx

                Cc: Prashanth Nednoor

                Subject: Re:  RBD caching on 4K
                reads???
          
        
        Hi Bruce,

          hmm, sounds for me like the rbd cache.

          Can you look, if the cache is realy disabled in the running
          config with

          
          ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show
          | grep cache

          
          Udo
        
          On 30.01.2015 21:51, Bruce McFarland
            wrote:
        
        
          I
              have a cluster and have created a rbd device - /dev/rbd1.
              It shows up as expected with ‘rbd –image test info’ and
              rbd showmapped. I have been looking at cluster performance
              with the usual Linux block device tools – fio and vdbench.
              When I look at writes and large block sequential reads I’m
              seeing what I’d expect with performance limited by either
              my cluster interconnect bandwidth or the backend device
              throughput speeds – 1 GE frontend and cluster network and
              7200rpm SATA OSDs with 1 SSD/osd for journal. Everything
              looks good EXCEPT 4K random reads. There is caching
              occurring somewhere in my system that I haven’t been able
              to detect and suppress - yet.
            
           
          I’ve
              set ‘rbd_cache=false’ in the [client] section of ceph.conf
              on the client, monitor, and storage nodes. I’ve flushed
              the system caches on the client and storage nodes before
              test run ie vm.drop_caches=3 and set the huge pages to the
              maximum available to consume free system memory so that it
              can’t be used for system cache . I’ve also disabled
              read-ahead on all of the HDD/OSDs.
           
          When
              I run a 4k randon read workload on the client the most I
              could expect would be ~100iops/osd x number of osd’s – I’m
              seeing an order of magnitude greater than that AND running
              IOSTAT on the storage nodes show no read activity on the
              OSD disks.
            
           
          Any
              ideas on what I’ve overlooked? There appears to be some
              read-ahead caching that I’ve missed.
            
           
          Thanks,
          Bruce
          

          _______________________________________________
          ceph-users mailing list
          ceph-users@xxxxxxxxxxxxxx
          http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
        
         
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com