cephfs quotas reporting

Goncalo Borges <goncalo.borges@xxxxxxxxxxxxx> · Mon, 5 Dec 2016 14:27:04 +1100



    Hi Again...

    
    Once more, my environment: 

    - ceph/cephfs in 10.2.2. 

      - All infrastructure is in the same version (rados cluster, mons,
      mds and cephfs clients). 

      - We mount cephfs using ceph-fuse.

    
    I want to set up quotas to limit users from filling the filesystem
    and proactively avoid a situation where I have several simultaneous
    full or near full osds. However, I am not understanding how the
    reporting of space works once quotas are in place. My cephfs cluster
    provides a ~100TB of space (~ 300 TB of raw space since I have a
    replication of 3x). Check the following two cases:

    
    1./ In clients where the full filesystem hierarchy is
      available: 

      
      - I have the following quota:

        # getfattr -n ceph.quota.max_bytes /coepp/cephfs

          getfattr: Removing leading '/' from absolute path names

          # file: coepp/cephfs

          ceph.quota.max_bytes="88000000000000"

        
        - I am mounting the client as

        # ceph-fuse --id mount_user -k
          /etc/ceph/ceph.client.mount_user.keyring -m <MON
          IP>:6789 --client-quota
          --fuse_default_permissions=0 --client_acl_type=posix_acl -r
          /cephfs /coepp/cephfs/

          
        - The results of two consecutive 'df' commands, executed after
        the mount operation, is the following. You can conclude that in
        the first command, the reported values are computed with respect
        to the quota but then fallback to the default when no quotas is
        in place.  

        
        #  puppet agent -t; df -h ; df -h

          (...)

          ceph-fuse              81T   51T   30T  64% /coepp/cephfs

          (...)

          ceph-fuse             306T  153T  154T  50% /coepp/cephfs

        
    2./ On another type of clients where I only mount a subdirectory of
    the filesystem (/coepp/cephfs/borg instead of coepp/cephfs):

    
      - I have the following quota:

        
        # getfattr -n ceph.quota.max_bytes
          /coepp/cephfs/borg

          getfattr: Removing leading '/' from absolute path name

          # file: coepp/cephfs/borg

          ceph.quota.max_bytes="10000000000000"

        
        - I mount the filesystem as:

        
        # ceph-fuse --id mount_user -k
          /etc/ceph/ceph.client.mount_user.keyring -m <MON
          IP>:6789 --client-quota --fuse_default_permissions=0
          --client_acl_type=posix_acl -r /cephfs/borg /coepp/cephfs/borg

        
        - The reported space is

        
        # puppet agent -t; df -h ; df -h

          (...)

          ceph-fuse       9.1T  5.7T  3.5T  62% /coepp/cephfs/borg

          (...)

          ceph-fuse        81T   51T   30T  64% /coepp/cephfs/borg

        
    3./ Both clients are behaving in the same way where they start by
    reporting according to the implemented quota 

        
                51T of used space with respect to 81TB in total (case 1)

                5.7T of used space with respect to 9.1T in total (case
    2)

    
    and then fallback to the values enforced in the previous level of
    the hierarchy.

    
                153T of used space with respect to 306T in total (case
    1)

                51T of used space with respect to 81TB in total (case 2)

    
    Am i doing something wrong here?

    
    Cheers

    Goncalo

    -- 
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW  2006
T: +61 2 93511937
  

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com