On Mon, 10 Feb 2020, Gregory Farnum wrote:
On Mon, Feb 10, 2020 at 12:29 AM Håkan T Johansson <f96hajo@xxxxxxxxxxx> wrote:
On Mon, 10 Feb 2020, Gregory Farnum wrote:
On Sun, Feb 9, 2020 at 3:24 PM Håkan T Johansson <f96hajo@xxxxxxxxxxx> wrote:
Hi,
running 14.2.6, debian buster (backports).
Have set up a cephfs with 3 data pools and one metadata pool:
myfs_data, myfs_data_hdd, myfs_data_ssd, and myfs_metadata.
The data of all files are with the use of ceph.dir.layout.pool either
stored in the pools myfs_data_hdd or myfs_data_ssd. This has also been
checked by dumping the ceph.file.layout.pool attributes of all files.
The filesystem has 1617949 files and 36042 directories.
There are however approximately as many objects in the first pool created
for the cephfs, myfs_data, as there are files. They also becomes more or
fewer as files are created or deleted (so cannot be some leftover from
earlier exercises). Note how the USED size is reported as 0 bytes,
correctly reflecting that no file data is stored in them.
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
myfs_data 0 B 1618229 0 4854687 0 0 0 2263590 129 GiB 23312479 124 GiB 0 B 0 B
myfs_data_hdd 831 GiB 136309 0 408927 0 0 0 106046 200 GiB 269084 277 GiB 0 B 0 B
myfs_data_ssd 43 GiB 1552412 0 4657236 0 0 0 181468 2.3 GiB 4661935 12 GiB 0 B 0 B
myfs_metadata 1.2 GiB 36096 0 108288 0 0 0 4828623 82 GiB 1355102 143 GiB 0 B 0 B
Is this expected?
I was assuming that in this scenario, all objects, both their data and any
keys would be either in the metadata pool, or the two pools where the
objects are stored.
Is it some additional metadata keys that are stored in the first
created data pool for cephfs? This would not be so nice in case the osd
selection rules for it are using worse disks than the data itself...
https://docs.ceph.com/docs/master/cephfs/file-layouts/#adding-a-data-pool-to-the-mds notes there is “a small amount of metadata” kept in the primary pool.
Thanks! This I managed to miss, probably as it was at the bottom of the
page. In case one wants to use layouts to separate fast (likely many)
from slow (likely large) files, it then sounds as the primary pool should
the fast kind too, due to the large amount of objects. Thus this needs to
be highlighted early in that documentation.
That’s not terribly clear; what is actually stored is a per-file location backtrace (its location in the directory tree) used for hardlink lookups and disaster recovery
scenarios.
This info would be nice to add to the manual page. It is nice to know
what kind of information is stored there.
Yeah, PRs welcome. :p
Just to be clear though, that shouldn't be performance-critical. It's
lazily updated by the MDS when the directory location changes, but not
otherwise.
The sheer amount of objects seems to make a big difference when a pg is
rebalanced between drives though, in particular from a HDD (with SSD DB),
despite having no data ? (Also comparing to the metadata pool, which does
not have one object per file.)
Best regards,
Håkan
Again thanks for the clarification!
Btw: is there any tool to see the amount of key value data size associated
with a pool? 'ceph osd df' gives omap and meta for osds, but not broken
down per pool.
I think this is in the newest master code, but I’m not certain which release it’s in...
Would it then (when available) also be in the 'rados df' command?
I really don't remember how everything is shared out but I think so?
Best regards,
Håkan
-Greg
Best regards,
Håkan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
