2011/7/27 Sławomir Skowron <szibis@xxxxxxxxx>: > Ok, I will show example: > > rados df > pool name KB objects clones degraded > unfound rd rd KB wr wr KB > .log 558212 5 0 0 > 0 0 0 2844888 2844888 > .pool 1 1 0 0 > 0 0 0 8 8 > .rgw 0 6 0 0 > 0 0 0 1 0 > .users 1 1 0 0 > 0 0 0 1 1 > .users.email 1 1 0 0 > 0 0 0 1 1 > .users.uid 2 2 0 0 > 0 1 0 2 2 > data 0 0 0 0 > 0 0 0 0 0 > metadata 0 0 0 0 > 0 0 0 0 0 > rbd 0 0 0 0 > 0 0 0 0 0 > sstest 32244226 2841055 0 653353 > 0 0 0 17066724 32370391 > total used 324792996 2841071 > total avail 31083452176 > total space 33043244460 > > It means I have almost 3mln of objects in sstest. > > pg_pool 7 'sstest' pg_pool(rep pg_size 3 crush_ruleset 0 object_hash > rjenkins pg_num 8 pgp_num 8 lpg_num 0 lpgp_num 0 last_change 21 owner > 0) > > 3 copies in this pool. > > sstest used 32.244.226 KB + log 558.212 KB = 32.802.438 KB > > Total used is 324.792.996 KB and it's almost 10x more. > > 2011-07-27 12:57:35.541556 pg v54158: 6986 pgs: 8 active, 6978 > active+clean; 32104 MB data, 310 GB used, 29642 GB / 31512 GB avail; > > I'am putting files beetwen 4-50KB on RADOS via s3 clilent, and radosgw. > > > Can you explain that to me on this example from real life ?? Hmm, what underlying filesystem are you using? Do you have any logging enabled, and what disk is it logging to? Are all your OSDs running under the same OS, or are they in virtual machines? If I remember correctly, that "total used" count is generated by looking at df or something for the drives in question -- if there's other data on the same drive as the OSD, it'll get (admittedly incorrectly) counted as part of the "total used" by RADOS even if RADOS can't touch it. -Greg ˙ôčş{.nÇ+?ˇ?Ž??+%?Ë˙ąéÝśĽ?w˙ş{.nÇ+?ˇ?z?˙uëŢ?ř§ś?ĄÜ¨}Š?˛Ć zÚ&j:+v?¨ţřŻůŽwĽţ?ŕ2?Ţ?¨čÚ&˘)ߥŤaśÚ˙˙űŕzżäzšŢ?ú+?ů???ݢj˙?wčţf