...I bet the leveldb/omap stuff isn't being included in the of statistics. That could be why and would make sense with what you've got here. :)
-Greg
On Sat, Apr 25, 2015 at 10:32 AM Adam Tygart <mozes@xxxxxxxxxxx> wrote:
cephfs (really ec84pool) is an ec pool (k=8 m=4), cachepool is a
writeback cachetier in front of ec84pool. As far as I know, we've not
done any strange configuration.
Potentially relevant configuration details:
ceph osd crush dump >
http://people.beocat.cis.ksu.edu/~mozes/ceph/crush_dump.txt
ceph osd pool ls detail >
http://people.beocat.cis.ksu.edu/~mozes/ceph/pool_ls_detail.txt
ceph mds dump > http://people.beocat.cis.ksu.edu/~mozes/ceph/mds_dump.txt
getfattr -d -m '.*' /tmp/cephfs >
http://people.beocat.cis.ksu.edu/~mozes/ceph/getfattr_cephfs.txt
rsync is ongoing, moving data into cephfs. It would seem the data is
truly there, both with metadata and file data. md5sums match for files
that I've tested.
--
Adam
On Sat, Apr 25, 2015 at 12:16 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
> That doesn't make sense -- 50MB for 36 million files is <1.5 bytes each. How
> do you have things configured, exactly?
>
> On Sat, Apr 25, 2015 at 9:32 AM Adam Tygart <mozes@xxxxxxxxxxx> wrote:
>>
>> We're currently putting data into our cephfs pool (cachepool in front
>> of it as a caching tier), but the metadata pool contains ~50MB of data
>> for 36 million files. If that were an accurate estimation, we'd have a
>> metadata pool closer to ~140GB. Here is a ceph df detail:
>>
>> http://people.beocat.cis.ksu.edu/~mozes/ceph_df_detail.txt
>>
>> I'm not saying it won't get larger, I have no idea of the code behind
>> it. This is just what it happens to be for us.
>> --
>> Adam
>>
>>
>> On Sat, Apr 25, 2015 at 11:29 AM, François Lafont <flafdivers@xxxxxxx>
>> wrote:
>> > Thanks Greg and Steffen for your answer. I will make some tests.
>> >
>> > Gregory Farnum wrote:
>> >
>> >> Yeah. The metadata pool will contain:
>> >> 1) MDS logs, which I think by default will take up to 200MB per
>> >> logical MDS. (You should have only one logical MDS.)
>> >> 2) directory metadata objects, which contain the dentries and inodes
>> >> of the system; ~4KB is probably generous for each?
>> >
>> > So one file in the cephfs generates one inode of ~4KB in the
>> > "metadata" pool, correct? So that (number-of-files-in-cephfs) x 4KB
>> > gives me an (approximative) estimation of the amount of data in the
>> > "metadata" pool?
>> >
>> >> 3) Some smaller data structures about the allocated inode range and
>> >> current client sessions.
>> >>
>> >> The data pool contains all of the file data. Presumably this is much
>> >> larger, but it will depend on your average file size and we've not
>> >> done any real study of it.
>> >
>> > --
>> > François Lafont
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@xxxxxxxxxxxxxx
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
