On Wed, Feb 29, 2012 at 5:06 AM, Sławomir Skowron <slawomir.skowron@xxxxxxxxx> wrote: > > Ok, it's intentional. > > We are checking meta info about files, then, checking md5 of file > content. In parallel, updating object that have change, and then > archiving this objects in another key, and last thing is deleting > objects that expires. > > This happens over and over, because, this site is changing many times. > > Now i don't have any idea, how to workaround this problem, without > shutdown this app :( I looked at your osd log again, and there are other things that don't look right. I'll also need you to turn on 'debug osd = 20' and 'debug filestore = 20'. Other than that, I just pushed a workaround that might improve things. It's on the wip-rgw-atomic-no-retry branch on github (based on 0.42.2), so you might want to give it a spin and let us know whether it actually improved things. > > Another question is, why data from radosgw writing almost everything > only on one osd, and copies to 2 others, and only those. Is anyone can > explain this to me ?? > > /dev/sdu 275G 605M 260G 1% /vol0/data/osd.18 > /dev/sdw 275G 608M 260G 1% /vol0/data/osd.21 > /dev/sdz 275G 638M 260G 1% /vol0/data/osd.24 > /dev/sde 275G 605M 260G 1% /vol0/data/osd.3 > /dev/sdr 275G 605M 260G 1% /vol0/data/osd.16 > /dev/sdaa 275G 696M 260G 1% /vol0/data/osd.25 > /dev/sdp 275G 605M 260G 1% /vol0/data/osd.14 > /dev/sdd 275G 605M 260G 1% /vol0/data/osd.2 > /dev/sdk 275G 605M 260G 1% /vol0/data/osd.8 > /dev/sdh 275G 608M 260G 1% /vol0/data/osd.6 > /dev/sds 275G 605M 260G 1% /vol0/data/osd.17 > /dev/sdf 275G 638M 260G 1% /vol0/data/osd.4 > /dev/sdj 275G 637M 260G 1% /vol0/data/osd.9 > /dev/sdc 275G 604M 260G 1% /vol0/data/osd.1 > /dev/sdv 275G 2.7G 258G 2% /vol0/data/osd.20 > /dev/sdn 275G 607M 260G 1% /vol0/data/osd.12 > /dev/sdo 275G 605M 260G 1% /vol0/data/osd.13 > /dev/sdg 275G 605M 260G 1% /vol0/data/osd.5 > /dev/sdy 275G 633M 260G 1% /vol0/data/osd.23 > /dev/sdm 275G 605M 260G 1% /vol0/data/osd.11 > /dev/sdx 275G 605M 260G 1% /vol0/data/osd.22 > /dev/sdb 275G 608M 260G 1% /vol0/data/osd.0 > /dev/sdq 275G 605M 260G 1% /vol0/data/osd.15 > /dev/sdl 275G 605M 260G 1% /vol0/data/osd.10 > /dev/sdi 275G 605M 260G 1% /vol0/data/osd.7 > /dev/sdt 275G 604M 260G 1% /vol0/data/osd.19 > That's probably because you're overwriting the same object over and over. When you do that the old object is still being kept so that any pending readers may complete reading it. You should run a maintenance utility in order to remove those (radosgw-admin temp remove --date=.. to remove all temp objects that were created before a specific date). Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
