2012/3/1 Sławomir Skowron <slawomir.skowron@xxxxxxxxx>: > 2012/2/29 Yehuda Sadeh Weinraub <yehuda.sadeh@xxxxxxxxxxxxx>: >> On Wed, Feb 29, 2012 at 5:06 AM, Sławomir Skowron >> <slawomir.skowron@xxxxxxxxx> wrote: >>> >>> Ok, it's intentional. >>> >>> We are checking meta info about files, then, checking md5 of file >>> content. In parallel, updating object that have change, and then >>> archiving this objects in another key, and last thing is deleting >>> objects that expires. >>> >>> This happens over and over, because, this site is changing many times. >>> >>> Now i don't have any idea, how to workaround this problem, without >>> shutdown this app :( >> >> I looked at your osd log again, and there are other things that don't >> look right. I'll also need you to turn on 'debug osd = 20' and 'debug >> filestore = 20'. > > osd.24 almost 10 minutes of log in debug, as above in attachment. > >> Other than that, I just pushed a workaround that might improve things. >> It's on the wip-rgw-atomic-no-retry branch on github (based on >> 0.42.2), so you might want to give it a spin and let us know whether >> it actually improved things. > > Ok i will try, and let you know soon. Unfortunately, no improvment after upgrade for this version. Ningx reports growing time for radosgw backend, and after increase this time to 33 seconds, changing into timeout. 10.177.8.11 - - - [05/Mar/2012:10:14:10 +0100] "PUT /test/_x/sidebar2/_d/widgets/sym/button_bw.png HTTP/1.1" rlength: 1049 bsent: 232 rtime: 2.487 urtime: 2.485 status: 200 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.12 - - - [05/Mar/2012:10:14:10 +0100] "PUT /test/_x/sidebar2/_d/widgets/sym/button_bw.png?acl HTTP/1.1" rlength: 353 bsent: 157 rtime: 0.013 urtime: 0.013 status: 200 bbsent: 5 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:13 +0100] "PUT /test/_x/sidebar2/_d/widgets/newmail/list.png?acl HTTP/1.1" rlength: 352 bsent: 157 rtime: 0.006 urtime: 0.006 status: 200 bbsent: 5 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:17 +0100] "PUT /test/_x/sidebar2/_d/widgets/zumisearch/button.png?acl HTTP/1.1" rlength: 357 bsent: 157 rtime: 0.006 urtime: 0.006 status: 200 bbsent: 5 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:21 +0100] "PUT /test/_g/sidebar/_d/widgets/globaltime/city_point_green.gif HTTP/1.1" rlength: 1297 bsent: 232 rtime: 13.427 urtime: 13.426 status: 200 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:26 +0100] "PUT /test/_x/sidebar2/_d/widgets/zum/zumikombw.png HTTP/1.1" rlength: 4278 bsent: 232 rtime: 18.384 urtime: 18.383 status: 200 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.12 - - - [05/Mar/2012:10:14:29 +0100] "PUT /test/_x/sidebar2/_d/widgets/allegro/allegro.png HTTP/1.1" rlength: 4129 bsent: 232 rtime: 21.963 urtime: 21.962 status: 200 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:32 +0100] "PUT /test/_g/sidebar/_d/popover/mdcbtn.gif HTTP/1.1" rlength: 1532 bsent: 232 rtime: 25.105 urtime: 25.104 status: 200 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.12 - - - [05/Mar/2012:10:14:40 +0100] "PUT /test/40622%2Cserwisy.html HTTP/1.1" rlength: 18178 bsent: 232 rtime: 32.651 urtime: 32.648 status: 200 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.12 - - - [05/Mar/2012:10:14:40 +0100] "PUT /test/_x/sidebar/cssb/popover/btn-nrm.gif HTTP/1.1" rlength: 539 bsent: 25 rtime: 33.003 urtime: - status: 499 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:40 +0100] "PUT /test/DaneAjax%2CWiadomosciLokalne%2Cajax.json%3Fmiasto%3Dslask%26z%3D0%26v%3D201203051014 HTTP/1.1" rlength: 1572 bsent: 25 rtime: 33.002 urtime: - status: 499 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.12 - - - [05/Mar/2012:10:14:40 +0100] "PUT /test/37828%2Cserwisy.html HTTP/1.1" rlength: 18363 bsent: 25 rtime: 33.003 urtime: - status: 499 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:40 +0100] "PUT /test/DaneAjax%2CWiadomosciLokalne%2Cajax.json%3Fmiasto%3Dpoznan%26z%3D0%26v%3D201203051014 HTTP/1.1" rlength: 1602 bsent: 25 rtime: 33.004 urtime: - status: 499 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:40 +0100] "PUT /test/DaneAjax%2CWiadomosciLokalne%2Cajax.json%3Fmiasto%3Dbydgoszcz%26z%3D0%26v%3D201203051014 HTTP/1.1" rlength: 1738 bsent: 25 rtime: 33.005 urtime: - status: 499 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" 10.177.8.11 - - - [05/Mar/2012:10:14:40 +0100] "PUT /test/_x/sidebar2/_d/widgets/sym/button.png HTTP/1.1" rlength: 1398 bsent: 25 rtime: 33.001 urtime: - status: 499 bbsent: 25 httpref: "-" useragent: "Boto/2.2.2 (linux2)" > >> >> >>> >>> Another question is, why data from radosgw writing almost everything >>> only on one osd, and copies to 2 others, and only those. Is anyone can >>> explain this to me ?? >>> >>> /dev/sdu 275G 605M 260G 1% /vol0/data/osd.18 >>> /dev/sdw 275G 608M 260G 1% /vol0/data/osd.21 >>> /dev/sdz 275G 638M 260G 1% /vol0/data/osd.24 >>> /dev/sde 275G 605M 260G 1% /vol0/data/osd.3 >>> /dev/sdr 275G 605M 260G 1% /vol0/data/osd.16 >>> /dev/sdaa 275G 696M 260G 1% /vol0/data/osd.25 >>> /dev/sdp 275G 605M 260G 1% /vol0/data/osd.14 >>> /dev/sdd 275G 605M 260G 1% /vol0/data/osd.2 >>> /dev/sdk 275G 605M 260G 1% /vol0/data/osd.8 >>> /dev/sdh 275G 608M 260G 1% /vol0/data/osd.6 >>> /dev/sds 275G 605M 260G 1% /vol0/data/osd.17 >>> /dev/sdf 275G 638M 260G 1% /vol0/data/osd.4 >>> /dev/sdj 275G 637M 260G 1% /vol0/data/osd.9 >>> /dev/sdc 275G 604M 260G 1% /vol0/data/osd.1 >>> /dev/sdv 275G 2.7G 258G 2% /vol0/data/osd.20 >>> /dev/sdn 275G 607M 260G 1% /vol0/data/osd.12 >>> /dev/sdo 275G 605M 260G 1% /vol0/data/osd.13 >>> /dev/sdg 275G 605M 260G 1% /vol0/data/osd.5 >>> /dev/sdy 275G 633M 260G 1% /vol0/data/osd.23 >>> /dev/sdm 275G 605M 260G 1% /vol0/data/osd.11 >>> /dev/sdx 275G 605M 260G 1% /vol0/data/osd.22 >>> /dev/sdb 275G 608M 260G 1% /vol0/data/osd.0 >>> /dev/sdq 275G 605M 260G 1% /vol0/data/osd.15 >>> /dev/sdl 275G 605M 260G 1% /vol0/data/osd.10 >>> /dev/sdi 275G 605M 260G 1% /vol0/data/osd.7 >>> /dev/sdt 275G 604M 260G 1% /vol0/data/osd.19 >>> >> That's probably because you're overwriting the same object over and >> over. When you do that the old object is still being kept so that any >> pending readers may complete reading it. You should run a maintenance >> utility in order to remove those (radosgw-admin temp remove --date=.. >> to remove all temp objects that were created before a specific date). > > Nice, useful advice, thanks. > >> >> Yehuda > > -- > ----- > Pozdrawiam > > Sławek Skowron -- ----- Pozdrawiam Sławek "sZiBis" Skowron -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
