To answer my own question. It seems Frank Schilder asked a similar question two years ago: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/6ENI42ZMHTTP2OONBRD7FDP7LQBC4P2E/ listxattr() was aparrently removed and not much have happen since then it seems. Anyway, I just made my own ceph-fs version of "du". ceph_du_dir: #!/bin/bash # usage: ceph_du_dir $DIR SIZE=$(getfattr -n ceph.dir.rbytes $1 2>/dev/null| grep "ceph\.dir\.rbytes" | awk -F\= '{print $2}' | sed s/\"//g) numfmt --to=iec-i --suffix=B --padding=7 $SIZE Prints out ceph-fs dir size in "human-readble" It works like a charm and my god it is fast!..... Tools like that could be very useful, if provided by the development team 🙂 Best, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka@xxxxxxxxx Tlf: +45 50906203 ________________________________ Fra: Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx> Sendt: 16. december 2021 14:37 Til: Robert Gallop <robert.gallop@xxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Emne: Re: cephfs quota used Woops, wrong copy/pasta: getfattr -n ceph.dir.rbytes $DIR works on all distributions I have tested. It is: getfattr -d -m 'ceph.*' $DIR that does not work on Rocky Linux 8, Ubuntu 18.04, but works on CentOS 7. Best, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka@xxxxxxxxx Tlf: +45 50906203 ________________________________ Fra: Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx> Sendt: 16. december 2021 13:57 Til: Robert Gallop <robert.gallop@xxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Emne: Re: cephfs quota used Just tested: getfattr -n ceph.dir.rbytes $DIR Works on CentOS 7, but not on Ubuntu 18.04 eighter. Weird? Best, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka@xxxxxxxxx Tlf: +45 50906203 ________________________________ Fra: Robert Gallop <robert.gallop@xxxxxxxxx> Sendt: 16. december 2021 13:42 Til: Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx> Cc: ceph-users@xxxxxxx <ceph-users@xxxxxxx> Emne: Re: Re: cephfs quota used >From what I understand you used to be able to do that but cannot on later kernels? Seems there would be a list somewhere, but I can’t find it, maybe it’s changing too often depending on the kernel your using or something. But yeah, these attrs are one of the major reasons we are moving from traditional appliance NAS to ceph, the many other benefits come with it. On Thu, Dec 16, 2021 at 5:38 AM Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx<mailto:jelka@xxxxxxxxx>> wrote: Thanks everybody, That was a quick answer. getfattr -n ceph.dir.rbytes $DIR Was the answer that worked for me. So getfattr was the solution after all. Is there some way I can display all attributes, without knowing them in forehand? I have tried: getfattr -d -m 'ceph.*' $DIR which gives me no output. Should that not list all atributes? This is on Rocky Linux kernel 4.18.0-348.2.1.el8_5.x86_64 Best, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka@xxxxxxxxx<mailto:jelka@xxxxxxxxx> Tlf: +45 50906203 ________________________________ Fra: Sebastian Knust <sknust@xxxxxxxxxxxxxxxxxxxxxxx<mailto:sknust@xxxxxxxxxxxxxxxxxxxxxxx>> Sendt: 16. december 2021 13:01 Til: Jesper Lykkegaard Karlsen <jelka@xxxxxxxxx<mailto:jelka@xxxxxxxxx>>; ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> <ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>> Emne: Re: cephfs quota used Hi Jasper, On 16.12.21 12:45, Jesper Lykkegaard Karlsen wrote: > Now, I want to access the usage information of folders with quotas from root level of the cephfs. > I have failed to find this information through getfattr commands, only quota limits are shown here, and du-command on individual folders is a suboptimal solution. `getfattr -n ceph.quota.max_bytes /path` gives the specified quota for a given path. `getfattr -n ceph.dir.rbytes /path` gives the size of the path, as you would usually get with du for conventional file systems. As an example, I am using this script for weekly utilisation reports: > for i in /ceph-path-to-home-dirs/*; do > if [ -d "$i" ]; then > SIZE=$(getfattr -n ceph.dir.rbytes --only-values "$i") > QUOTA=$(getfattr -n ceph.quota.max_bytes --only-values "$i" 2>/dev/null || echo 0) > PERC=$(echo $SIZE*100/$QUOTA | bc 2> /dev/null) > if [ -z "$PERC" ]; then PERC="--"; fi > printf "%-30s %8s %8s %8s%%\n" "$i" `numfmt --to=iec $SIZE` `numfmt --to=iec $QUOTA` $PERC > fi > done Note that you can also mount CephFS with the "rbytes" mount option. IIRC the fuse clients defaults to it, for the kernel client you have to specify it in the mount command or fstab entry. The rbytes option returns the recursive path size (so the ceph.dir.rbytes fattr) in stat calls to directories, so you will see it with ls immediately. I really like it! Just beware that some software might have issues with this behaviour - alpine is the only example (bug report and patch proposal have been submitted) that I know of. Cheers Sebastian _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |