On Tue, Sep 3, 2013 at 9:13 AM, Fuchs, Andreas (SwissTXT) <Andreas.Fuchs@xxxxxxxxxxx> wrote: > We are testing radosgw with cyberduck, so far we see the following issues > > 1. In apache error log for each file put we see: > > [Tue Sep 03 17:35:24 2013] [warn] FastCGI: 193.218.104.138 PUT https://193.218.100.131/test/tesfile04.iso auth AWS *** > [Tue Sep 03 17:35:24 2013] [warn] FastCGI: JJJ gotCont=1 > > - This is with the ceph version of apache/fastcgi. > - This happens on each file, 1 time at the beginning of the transfer and at least at large files, several times at the end. > - The standard apache/fastcgi combo does not show this issue. > About what do we get warned here? > That's a debug output that I thought was long gone. We need to remove it. > > 2. Large file transfer of 4GB iso image with cyberduck > Uploads of a 4GB .iso file fail at the end of each upload, the file is then shown with 0B in directory listing > > At the end of the upload I get the following in rados log > 2013-09-03 18:00:49.427092 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:01:11.427201 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:01:33.427310 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:01:36.557121 7faf46ff5700 10 x>> x-amz-acl:public-read > 2013-09-03 18:01:36.557567 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:tesfile05.iso state=0x7faf5412ecd8 s->prefetch_data=0 > 2013-09-03 18:01:36.558581 7faf46ff5700 0 setting object write_tag=default.6641.36 > 2013-09-03 18:01:36.574302 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1 state=0x7faf54028488 s->prefetch_data=0 > 2013-09-03 18:01:36.575215 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty > 2013-09-03 18:01:36.575250 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf54028488 > 2013-09-03 18:01:36.579593 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_2 state=0x7faf5412ea28 s->prefetch_data=0 > 2013-09-03 18:01:36.580581 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty > 2013-09-03 18:01:36.580635 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf5412ea28 > 2013-09-03 18:01:36.585058 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_3 state=0x7faf5412aaa8 s->prefetch_data=0 > 2013-09-03 18:01:36.585956 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty > 2013-09-03 18:01:36.586005 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf5412aaa8 > 2013-09-03 18:01:36.589811 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_4 state=0x7faf5412a068 s->prefetch_data=0 > 2013-09-03 18:01:36.590886 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty > 2013-09-03 18:01:36.590921 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf5412a068 > 2013-09-03 18:01:36.594977 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_5 state=0x7faf54126468 s->prefetch_data=0 > 2013-09-03 18:01:36.596140 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty > . > . > hunderts of similar lines > . > . > . > 2013-09-03 18:01:42.210941 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1038 state=0x7faf54172738 s->prefetch_data=0 > 2013-09-03 18:01:42.211888 7faf46ff5700 20 get_obj_state: s->obj_tag was set empty > 2013-09-03 18:01:42.211947 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf54172738 > 2013-09-03 18:01:42.216903 7faf46ff5700 20 get_obj_state: rctx=0x7faf540251c0 obj=test-han:_shadow__Bw6LkKRYSmAwEh5VBmxo7fb5Md-IY7d_1039 state=0x7faf54172a88 s->prefetch_data=0 > 2013-09-03 18:01:42.217757 7faf46ff5700 20 prepare_atomic_for_write_impl: state is not atomic. state=0x7faf54172a88 > 2013-09-03 18:01:42.221839 7faf46ff5700 0 WARNING: set_req_state_err err_no=27 resorting to 500 error 27 means EFBIG. I do see a few newish error paths in the osd that may return this. Can you try setting the following on your osd: osd max attr size = 655360 > 2013-09-03 18:01:42.221868 7faf46ff5700 2 req 36:402.848669:s3:PUT /test-han/tesfile05.iso:put_obj:http status=500 > 2013-09-03 18:01:42.223232 7faf46ff5700 1 ====== req done req=0x25e24b0 http_status=500 ====== > 2013-09-03 18:01:55.427440 7faf87fff700 2 RGWDataChangesLog::ChangesRenewThread: start > 2013-09-03 18:02:01.331323 7faf9d011780 20 enqueued request req=0x25e2020 > 2013-09-03 18:02:01.331341 7faf9d011780 20 RGWWQ: > 2013-09-03 18:02:01.331343 7faf9d011780 20 req: 0x25e2020 > 2013-09-03 18:02:01.331352 7faf9d011780 10 allocated request req=0x25e2500 > 2013-09-03 18:02:01.331377 7faf307c8700 20 dequeued request req=0x25e2020 > 2013-09-03 18:02:01.331385 7faf307c8700 20 RGWWQ: empty > > > > > > > 3. performance > While I get great performance from the radosgw to the stoage nodes, the performance from cyberduck to the radosgw is poor (10 times slower) > > On a rados filesystem mounted on the radosgw I get: arround 100MB/s write troughput (here I know it's the 1Gb sync network between the nodes who is the limiting factor). > With a cyberduck test I only get 10MB/s and I cannot identify the bottleneck, network between testclient and radosgw is 1Gb and verified. Can you verify that you have enough pgs per pool? Also, turning off the debug log might help some: debug rgw = 0 There are a few other logs that might affect, but I think that at this point everything is already turned off by default. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |