Hello,
I got a weird upload issue with Ceph - dumpling (0.67.2) and I don't
know if someone can help me out to pin point my problem...
Basically, If I'm trying to upload a 1Gb file, as soon as my upload is
completed, apache return a 500 error... no problem if I upload a 900Mb
file or less, just got that specific problem with any file bigger than
1Gb !!
I also got 2 apache server in place, one with the modify fastcgi module
and the other one without - both server generate the same issue/behavior...
One thing that I remark : with a 1Gb file or up, radosgw throw me a
bunch of these errors : ( not getting these with a 900Mb or less file
size... )
2013-08-27 11:18:09.850700 7f1e490ec700 20 get_obj_state: s->obj_tag was
set empty
2013-08-27 11:18:09.850705 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic. state=0x7f1dc40ad938
2013-08-27 11:18:09.859304 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0 obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_435
state=0x7f1dc40adbf8 s->prefetch_data=0
(...)
2013-08-27 11:18:09.878697 7f1e490ec700 0 WARNING: set_req_state_err
err_no=27 resorting to 500
not seeing anything others errors, no OSD errors, no MDS errors... Can't
find anything on google related to this warning msg: set_req_state_err
err_no=27
any clue ??
Thanks
M-A
= = =
ceph -v
ceph version 0.67.2 (eb4380dd036a0b644c6283869911d615ed729ac8)
ceph status
cluster eb16413a-xxxx-xxxx-xxxx-f23fddd6a5f6
health HEALTH_OK
monmap e1: 2 mons at
{coe-w1-stor-db01=10.150.2.101:6789/0,coe-w1-stor-db02=10.150.2.102:6789/0},
election epoch 30, quorum 0,1 coe-w1-stor-db01,coe-w1-stor-db02
osdmap e518: 101 osds: 101 up, 101 in
pgmap v8991: 288 pgs: 288 active+clean; 2633 MB data, 11312 MB
used, 250 TB / 250 TB avail
mdsmap e64: 1/1/1 up {0=coe-w1-stor-db02=up:active}, 1 up:standby
apache:
172.16.11.118 - - [27/Aug/2013:11:15:21 -0400] "GET
/aaa/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1" 200 1596 "-"
"Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:15:23 -0400] "GET / HTTP/1.1" 200 1673
"-" "Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:15:23 -0400] "GET
/aaa/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1" 200 573 "-"
"Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:15:23 -0400] "PUT
/aaa/XenServer-6.2-binpkg.iso HTTP/1.1" 500 377 "-" "Cyberduck/4.3.1
(Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:18:09 -0400] "PUT
/aaa/XenServer-6.2-binpkg.iso HTTP/1.1" 200 315 "-" "Cyberduck/4.3.1
(Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:18:27 -0400] "GET
/aaa/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1" 200 944 "-"
"Cyberduck/4.3.1 (Mac OS X/10.8.4) (i386)"
172.16.11.118 - - [27/Aug/2013:11:18:27 -0400] "HEAD
/aaa/XenServer-6.2-binpkg.iso HTTP/1.1" 404 248 "-" "Cyberduck/4.3.1
(Mac OS X/10.8.4) (i386)"
radosgw.log
2013-08-27 11:15:23.634295 7f1e490ec700 2 req 4:0.000457:s3:PUT
/aaa/XenServer-6.2-binpkg.iso:put_obj:verifying op params
2013-08-27 11:15:23.634297 7f1e490ec700 2 req 4:0.000459:s3:PUT
/aaa/XenServer-6.2-binpkg.iso:put_obj:executing
2013-08-27 11:15:31.797706 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:15:53.797852 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:16:15.798007 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:16:37.798146 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:16:59.798282 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:17:21.798415 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:17:43.798551 7f1e42ffd700 2
RGWDataChangesLog::ChangesRenewThread: start
2013-08-27 11:18:05.495773 7f1e490ec700 10 x>> x-amz-acl:private
2013-08-27 11:18:05.495932 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0 obj=aaa:XenServer-6.2-binpkg.iso
state=0x7f1dc405f3a8 s->prefetch_data=0
2013-08-27 11:18:05.497367 7f1e490ec700 0 setting object
write_tag=default.15804.4
2013-08-27 11:18:05.508505 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0 obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_1
state=0x7f1dc4037f88 s->prefetch_data=0
2013-08-27 11:18:05.509832 7f1e490ec700 20 get_obj_state: s->obj_tag was
set empty
(........ bunch of those messages ........ )
2013-08-27 11:18:09.840315 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic. state=0x7f1dc40ad4a8
2013-08-27 11:18:09.849362 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0 obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_434
state=0x7f1dc40ad938 s->prefetch_data=0
2013-08-27 11:18:09.850700 7f1e490ec700 20 get_obj_state: s->obj_tag was
set empty
2013-08-27 11:18:09.850705 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic. state=0x7f1dc40ad938
2013-08-27 11:18:09.859304 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0 obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_435
state=0x7f1dc40adbf8 s->prefetch_data=0
2013-08-27 11:18:09.860641 7f1e490ec700 20 get_obj_state: s->obj_tag was
set empty
2013-08-27 11:18:09.860646 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic. state=0x7f1dc40adbf8
2013-08-27 11:18:09.870525 7f1e490ec700 20 get_obj_state:
rctx=0x7f1dc40028b0 obj=aaa:_shadow__ivTDZIBHCbNVE4p-DDeXDJHni8SArBZ_436
state=0x7f1dc40ba0c8 s->prefetch_data=0
2013-08-27 11:18:09.871788 7f1e490ec700 20
prepare_atomic_for_write_impl: state is not atomic. state=0x7f1dc40ba0c8
2013-08-27 11:18:09.878697 7f1e490ec700 0 WARNING: set_req_state_err
err_no=27 resorting to 500
2013-08-27 11:18:09.878736 7f1e490ec700 2 req 4:166.244898:s3:PUT
/aaa/XenServer-6.2-binpkg.iso:put_obj:http status=500
2013-08-27 11:18:09.879430 7f1e490ec700 1 ====== req done req=0x167fc10
http_status=500 ======
2013-08-27 11:18:27.706950 7f1e6388d780 20 enqueued request req=0x1688170
2013-08-27 11:18:27.706961 7f1e6388d780 20 RGWWQ:
2013-08-27 11:18:27.706963 7f1e6388d780 20 req: 0x1688170
2013-08-27 11:18:27.707152 7f1e6388d780 10 allocated request req=0x162b780
2013-08-27 11:18:27.707220 7f1e490ec700 20 dequeued request req=0x1688170
2013-08-27 11:18:27.707230 7f1e490ec700 20 RGWWQ: empty
2013-08-27 11:18:27.707252 7f1e490ec700 1 ====== starting new
ceph.conf - with modify fastcgi
fsid = eb16413a-xxxx-xxxx-xxxx-f23fddd6a5f6
mon_initial_members = coe-w1-stor-db01, coe-w1-stor-db02
mon_host = 10.150.2.101,10.150.2.102
osd_journal_size = 253952
filestore_xattr_use_omap = true
osd_crush_chooseleaf_type = 1
rgw print continue = true
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
[client.radosgw.gateway]
host = coe-w1-stor-db01
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log
rgw_enable_usage_log = true
rgw_usage_log_tick_interval = 30
rgw_usage_log_flush_threshold = 1024
rgw_usage_max_shards = 32
rgw_usage_max_user_shards = 1
rgw_dns_name = storage.myserver.net
rgw_print_continue = true
ceph.conf - withiout modify fastcgi...
[global]
fsid = eb16413a-xxxx-xxxx-xxxx-f23fddd6a5f6
mon_initial_members = coe-w1-stor-db01, coe-w1-stor-db02
mon_host = 10.150.2.101,10.150.2.102
auth_supported = cephx
osd_journal_size = 253952
filestore_xattr_use_omap = true
osd_crush_chooseleaf_type = 1
rgw print continue = false
#auth_cluster_required = cephx
#auth_service_required = cephx
#auth_client_required = cephx
[client.radosgw.gateway]
host = coe-w1-stor-db02
keyring = /etc/ceph/keyring.radosgw.gateway
rgw_socket_path = /tmp/radosgw.sock
log_file = /var/log/ceph/radosgw.log
rgw_enable_usage_log = true
rgw_usage_log_tick_interval = 30
rgw_usage_log_flush_threshold = 1024
rgw_usage_max_shards = 32
rgw_usage_max_user_shards = 1
rgw_dns_name = storage.myserver.net
rgw_print_continue = false
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
