Re: CephFS vs RBD

Lionel Bouton <lionel+ceph@xxxxxxxxxxx> · Wed, 22 Jul 2015 21:51:43 +0200



    Le 22/07/2015 21:17, Lincoln Bryant a
      écrit :

    
      Hi Hadi,
      

      AFAIK, you can’t safely mount RBD as R/W on multiple
        machines. You could re-export the RBD as NFS, but that’ll
        introduce a bottleneck and probably tank your performance gains
        over CephFS.
      

      For what it’s worth, some of our RBDs are mapped to
        multiple machines, mounted read-write on one and read-only on
        the others. We haven’t seen any strange effects from that, but I
        seem to recall it being ill advised. 

      
    Yes it is, for several reasons. Here are two at the top of my head.

    
    Some (many/most/all?) filesystems update on-disk data when they are
    mounted even if the mount is read-only. If you map your RBD devices
    read-only before mounting the filesystem itself read-only you should
    be safe from corruption occurring at mount time though.

    The system with read-write access will keep its in-memory data in
    sync with the on-disk data. The others with read-only access will
    not as they won't be aware of the writes done, this means they will
    eventually get incoherent data and will generate fs access errors
    with various levels of errors from the benign read error to
    potentially full kernel crash with whole filesystem freeze
    in-between.

    
    Don't do that unless you :

    - carefully setup your rbd mappings read-only everywhere but the
    system doing the writes,

    - can withstand a (simultaneous) system crash on all the systems
    mounting the rbd mappings read-only.

    
    Lionel

  
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com