On Mon, May 27, 2019 at 6:54 PM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: > > Am 27.05.19 um 12:48 schrieb Oliver Freyermuth: > > Am 27.05.19 um 11:57 schrieb Dan van der Ster: > >> On Mon, May 27, 2019 at 11:54 AM Oliver Freyermuth > >> <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: > >>> > >>> Dear Dan, > >>> > >>> thanks for the quick reply! > >>> > >>> Am 27.05.19 um 11:44 schrieb Dan van der Ster: > >>>> Hi Oliver, > >>>> > >>>> We saw the same issue after upgrading to mimic. > >>>> > >>>> IIRC we could make the max_bytes xattr visible by touching an empty > >>>> file in the dir (thereby updating the dir inode). > >>>> > >>>> e.g. touch /cephfs/user/freyermu/.quota; rm /cephfs/user/freyermu/.quota > >>> > >>> sadly, no, not even with sync's in between: > >>> ----- > >>> $ touch /cephfs/user/freyermu/.quota; sync; rm -f /cephfs/user/freyermu/.quota; sync; getfattr --absolute-names --only-values -n ceph.quota.max_bytes /cephfs/user/freyermu/ > >>> /cephfs/user/freyermu/: ceph.quota.max_bytes: No such attribute > >>> ----- > >>> Also restarting the FUSE client after that does not change it. Maybe this requires the rest of the cluster to be upgraded to work? > >>> I'm just guessing here, but maybe the MDS needs the file creation / update of the directory inode to "update" the way the quota attributes are exported. If something changed here with Mimic, > >>> this would explain why the "touch" is needed. And this would also explain why this might only help if the MDS is upgraded to Mimic, too. > >>> > >> > >> I think the relevant change which is causing this is the new_snaps in mimic. > >> > >> Did you already enable them? `ceph fs set cephfs allow_new_snaps 1` > > > > Good point! We wanted to enable these anyways with Mimic. > > > > I've enabled it just now (since servers are still Luminous, that required "--yes-i-really-mean-it") but sadly, the max_bytes attribute is still not there > > (also not after remounting on the client / using the file creation and deletion trick). > > That's interesting - it suddenly started to work for one directory after creating a snapshot for one directory subtree on which we have quotas enabled, > and removing that snapshot again. > I can reproduce that for other directories. > So it seems enabling snapshots and snapshotting once fixes it for that directory tree. > > If that's the case, maybe this could be added to the upgrade notes? > quota handling code changed in mimic. mimic client + luminous mds have compat issue. there should be no issue if both mds and client are both upgraded to mimic, Regards Yan, Zheng > Cheers, > Oliver > > > > > Cheers, > > Oliver > > > >> > >> -- dan > >> > >> > >>> We have scheduled the remaining parts of the upgrade for Wednesday, and worst case could survive until then without quota enforcement, but it's a really strange and unexpected incompatibility. > >>> > >>> Cheers, > >>> Oliver > >>> > >>>> > >>>> Does that work? > >>>> > >>>> -- dan > >>>> > >>>> > >>>> On Mon, May 27, 2019 at 11:36 AM Oliver Freyermuth > >>>> <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: > >>>>> > >>>>> Dear Cephalopodians, > >>>>> > >>>>> in the process of migrating a cluster from Luminous (12.2.12) to Mimic (13.2.5), we have upgraded the FUSE clients first (we took the chance during a time of low activity), > >>>>> thinking that this should not cause any issues. All MDS+MON+OSDs are still on Luminous, 12.2.12. > >>>>> > >>>>> However, it seems quotas have stopped working - with a (FUSE) Mimic client (13.2.5), I see: > >>>>> $ getfattr --absolute-names --only-values -n ceph.quota.max_bytes /cephfs/user/freyermu/ > >>>>> /cephfs/user/freyermu/: ceph.quota.max_bytes: No such attribute > >>>>> > >>>>> A Luminous client (12.2.12) on the same cluster sees: > >>>>> $ getfattr --absolute-names --only-values -n ceph.quota.max_bytes /cephfs/user/freyermu/ > >>>>> 5000000000000 > >>>>> > >>>>> It does not seem as if the attribute has been renamed (e.g. https://github.com/ceph/ceph/blob/mimic/qa/tasks/cephfs/test_quota.py still references it, same for the docs), > >>>>> and I have to assume the clients also do not enforce quota if they do not see it. > >>>>> > >>>>> Is this a known incompatibility between Mimic clients and a Luminous cluster? > >>>>> The release notes of Mimic only mention that quota support was added to the kernel client, but nothing else quota related catches my eye. > >>>>> > >>>>> Cheers, > >>>>> Oliver > >>>>> > >>>>> _______________________________________________ > >>>>> ceph-users mailing list > >>>>> ceph-users@xxxxxxxxxxxxxx > >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >>> > >>> -- > >>> Oliver Freyermuth > >>> Universität Bonn > >>> Physikalisches Institut, Raum 1.047 > >>> Nußallee 12 > >>> 53115 Bonn > >>> -- > >>> Tel.: +49 228 73 2367 > >>> Fax: +49 228 73 7869 > >>> -- > >>> > > > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > -- > Oliver Freyermuth > Universität Bonn > Physikalisches Institut, Raum 1.047 > Nußallee 12 > 53115 Bonn > -- > Tel.: +49 228 73 2367 > Fax: +49 228 73 7869 > -- > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |