On Mon, May 27, 2019 at 11:54 AM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: > > Dear Dan, > > thanks for the quick reply! > > Am 27.05.19 um 11:44 schrieb Dan van der Ster: > > Hi Oliver, > > > > We saw the same issue after upgrading to mimic. > > > > IIRC we could make the max_bytes xattr visible by touching an empty > > file in the dir (thereby updating the dir inode). > > > > e.g. touch /cephfs/user/freyermu/.quota; rm /cephfs/user/freyermu/.quota > > sadly, no, not even with sync's in between: > ----- > $ touch /cephfs/user/freyermu/.quota; sync; rm -f /cephfs/user/freyermu/.quota; sync; getfattr --absolute-names --only-values -n ceph.quota.max_bytes /cephfs/user/freyermu/ > /cephfs/user/freyermu/: ceph.quota.max_bytes: No such attribute > ----- > Also restarting the FUSE client after that does not change it. Maybe this requires the rest of the cluster to be upgraded to work? > I'm just guessing here, but maybe the MDS needs the file creation / update of the directory inode to "update" the way the quota attributes are exported. If something changed here with Mimic, > this would explain why the "touch" is needed. And this would also explain why this might only help if the MDS is upgraded to Mimic, too. > I think the relevant change which is causing this is the new_snaps in mimic. Did you already enable them? `ceph fs set cephfs allow_new_snaps 1` -- dan > We have scheduled the remaining parts of the upgrade for Wednesday, and worst case could survive until then without quota enforcement, but it's a really strange and unexpected incompatibility. > > Cheers, > Oliver > > > > > Does that work? > > > > -- dan > > > > > > On Mon, May 27, 2019 at 11:36 AM Oliver Freyermuth > > <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: > >> > >> Dear Cephalopodians, > >> > >> in the process of migrating a cluster from Luminous (12.2.12) to Mimic (13.2.5), we have upgraded the FUSE clients first (we took the chance during a time of low activity), > >> thinking that this should not cause any issues. All MDS+MON+OSDs are still on Luminous, 12.2.12. > >> > >> However, it seems quotas have stopped working - with a (FUSE) Mimic client (13.2.5), I see: > >> $ getfattr --absolute-names --only-values -n ceph.quota.max_bytes /cephfs/user/freyermu/ > >> /cephfs/user/freyermu/: ceph.quota.max_bytes: No such attribute > >> > >> A Luminous client (12.2.12) on the same cluster sees: > >> $ getfattr --absolute-names --only-values -n ceph.quota.max_bytes /cephfs/user/freyermu/ > >> 5000000000000 > >> > >> It does not seem as if the attribute has been renamed (e.g. https://github.com/ceph/ceph/blob/mimic/qa/tasks/cephfs/test_quota.py still references it, same for the docs), > >> and I have to assume the clients also do not enforce quota if they do not see it. > >> > >> Is this a known incompatibility between Mimic clients and a Luminous cluster? > >> The release notes of Mimic only mention that quota support was added to the kernel client, but nothing else quota related catches my eye. > >> > >> Cheers, > >> Oliver > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > Oliver Freyermuth > Universität Bonn > Physikalisches Institut, Raum 1.047 > Nußallee 12 > 53115 Bonn > -- > Tel.: +49 228 73 2367 > Fax: +49 228 73 7869 > -- > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |