Since you mention a problem starting at
1GB, check to see if you have a LimitRequestBody directive
( http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestbody)
The LimitRequestBody directive allows the
user to set a limit on the allowed size of an HTTP request
message body within the context in which the directive is
given.
To me, this doesn't look like an Apache problem. If it was
LimitRequestBody, Apache should deny the request, and RadosGW
would never see it.
But it's quick and easy to verify that this isn't a problem. I'd
check the other LimitRequest parameters too.
On 8/27/13 08:54 , Marc-Andre Jutras wrote:
Hello,
I got a weird upload issue with Ceph - dumpling (0.67.2) and I
don't know if someone can help me out to pin point my problem...
Basically, If I'm trying to upload a 1Gb file, as soon as my
upload is completed, apache return a 500 error... no problem if I
upload a 900Mb file or less, just got that specific problem with
any file bigger than 1Gb !!
I also got 2 apache server in place, one with the modify fastcgi
module and the other one without - both server generate the same
issue/behavior...
One thing that I remark : with a 1Gb file or up, radosgw throw me
a bunch of these errors : ( not getting these with a 900Mb or less
file size... )
2013-08-27 11:18:09.850700 7f1e490ec700 20 get_obj_state:
s->obj_tag was set empty
2013-08-27 11:18:09.850705 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic.
state=0x7f1dc40ad938
2013-08-27 11:18:09.859304 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0
obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_435
state=0x7f1dc40adbf8 s->prefetch_data=0
(...)
2013-08-27 11:18:09.878697 7f1e490ec700 0 WARNING:
set_req_state_err err_no=27 resorting to 500
not seeing anything others errors, no OSD errors, no MDS errors...
Can't find anything on google related to this warning msg:
set_req_state_err err_no=27
any clue ??
Thanks
M-A
= = =
ceph -v
ceph version 0.67.2 (eb4380dd036a0b644c6283869911d615ed729ac8)
ceph status
cluster eb16413a-xxxx-xxxx-xxxx-f23fddd6a5f6
health HEALTH_OK
monmap e1: 2 mons at
{coe-w1-stor-db01=10.150.2.101:6789/0,coe-w1-stor-db02=10.150.2.102:6789/0},
election epoch 30, quorum 0,1 coe-w1-stor-db01,coe-w1-stor-db02
osdmap e518: 101 osds: 101 up, 101 in
pgmap v8991: 288 pgs: 288 active+clean; 2633 MB data, 11312 MB
used, 250 TB / 250 TB avail
mdsmap e64: 1/1/1 up {0=coe-w1-stor-db02=up:active}, 1
up:standby
apache:
172.16.11.118 - - [27/Aug/2013:11:15:21 -0400] "GET
/aaa/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1" 200
1596 "-" "Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:15:23 -0400] "GET / HTTP/1.1"
200 1673 "-" "Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:15:23 -0400] "GET
/aaa/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1" 200 573
"-" "Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:15:23 -0400] "PUT
/aaa/XenServer-6.2-binpkg.iso HTTP/1.1" 500 377 "-"
"Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:18:09 -0400] "PUT
/aaa/XenServer-6.2-binpkg.iso HTTP/1.1" 200 315 "-"
"Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:18:27 -0400] "GET
/aaa/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1" 200 944
"-" "Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:18:27 -0400] "HEAD
/aaa/XenServer-6.2-binpkg.iso HTTP/1.1" 404 248 "-"
"Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
radosgw.log
2013-08-27 11:15:23.634295 7f1e490ec700 2 req 4:0.000457:s3:PUT
/aaa/XenServer-6.2-binpkg.iso:put_obj:verifying op params
2013-08-27 11:15:23.634297 7f1e490ec700 2 req 4:0.000459:s3:PUT
/aaa/XenServer-6.2-binpkg.iso:put_obj:executing
2013-08-27 11:15:31.797706 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:15:53.797852 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:16:15.798007 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:16:37.798146 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:16:59.798282 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:17:21.798415 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:17:43.798551 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:18:05.495773 7f1e490ec700 10 x>>
x-amz-acl:private
2013-08-27 11:18:05.495932 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0 obj=aaa:XenServer-6.2-binpkg.iso
state=0x7f1dc405f3a8 s->prefetch_data=0
2013-08-27 11:18:05.497367 7f1e490ec700 0 setting object
write_tag=default.15804.4
2013-08-27 11:18:05.508505 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0
obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_1
state=0x7f1dc4037f88 s->prefetch_data=0
2013-08-27 11:18:05.509832 7f1e490ec700 20 get_obj_state:
s->obj_tag was set empty
(........ bunch of those messages ........ )
2013-08-27 11:18:09.840315 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic.
state=0x7f1dc40ad4a8
2013-08-27 11:18:09.849362 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0
obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_434
state=0x7f1dc40ad938 s->prefetch_data=0
2013-08-27 11:18:09.850700 7f1e490ec700 20 get_obj_state:
s->obj_tag was set empty
2013-08-27 11:18:09.850705 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic.
state=0x7f1dc40ad938
2013-08-27 11:18:09.859304 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0
obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_435
state=0x7f1dc40adbf8 s->prefetch_data=0
2013-08-27 11:18:09.860641 7f1e490ec700 20 get_obj_state:
s->obj_tag was set empty
2013-08-27 11:18:09.860646 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic.
state=0x7f1dc40adbf8
2013-08-27 11:18:09.870525 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0
obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_436
state=0x7f1dc40ba0c8 s->prefetch_data=0
2013-08-27 11:18:09.871788 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic.
state=0x7f1dc40ba0c8
2013-08-27 11:18:09.878697 7f1e490ec700 0 WARNING:
set_req_state_err err_no=27 resorting to 500
2013-08-27 11:18:09.878736 7f1e490ec700 2 req 4:166.244898:s3:PUT
/aaa/XenServer-6.2-binpkg.iso:put_obj:http status=500
2013-08-27 11:18:09.879430 7f1e490ec700 1 ====== req done
req=0x167fc10 http_status=500 ======
2013-08-27 11:18:27.706950 7f1e6388d780 20 enqueued request
req=0x1688170
2013-08-27 11:18:27.706961 7f1e6388d780 20 RGWWQ:
2013-08-27 11:18:27.706963 7f1e6388d780 20 req: 0x1688170
2013-08-27 11:18:27.707152 7f1e6388d780 10 allocated request
req=0x162b780
2013-08-27 11:18:27.707220 7f1e490ec700 20 dequeued request
req=0x1688170
2013-08-27 11:18:27.707230 7f1e490ec700 20 RGWWQ: empty
2013-08-27 11:18:27.707252 7f1e490ec700 1 ====== starting new
ceph.conf - with modify fastcgi
fsid = eb16413a-xxxx-xxxx-xxxx-f23fddd6a5f6
mon_initial_members = coe-w1-stor-db01, coe-w1-stor-db02
mon_host = 10.150.2.101,10.150.2.102
osd_journal_size = 253952
filestore_xattr_use_omap = true
osd_crush_chooseleaf_type = 1
rgw print continue = true
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
[client.radosgw.gateway]
host = coe-w1-stor-db01
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log
rgw_enable_usage_log = true
rgw_usage_log_tick_interval = 30
rgw_usage_log_flush_threshold = 1024
rgw_usage_max_shards = 32
rgw_usage_max_user_shards = 1
rgw_dns_name = storage.myserver.net
rgw_print_continue = true
ceph.conf - withiout modify fastcgi...
[global]
fsid = eb16413a-xxxx-xxxx-xxxx-f23fddd6a5f6
mon_initial_members = coe-w1-stor-db01, coe-w1-stor-db02
mon_host = 10.150.2.101,10.150.2.102
auth_supported = cephx
osd_journal_size = 253952
filestore_xattr_use_omap = true
osd_crush_chooseleaf_type = 1
rgw print continue = false
#auth_cluster_required = cephx
#auth_service_required = cephx
#auth_client_required = cephx
[client.radosgw.gateway]
host = coe-w1-stor-db02
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log
rgw_enable_usage_log = true
rgw_usage_log_tick_interval = 30
rgw_usage_log_flush_threshold = 1024
rgw_usage_max_shards = 32
rgw_usage_max_user_shards = 1
rgw_dns_name = storage.myserver.net
rgw_print_continue = false
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
|
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
