Hi,
I have a strange problem with the rados gateway. I'm getting Http 411
status code (Missing Content Length) whenever I upload any file to ceph.
The setup is: ceph 0.94.5, ubuntu 14.04, tengine (patched nginx).
The strange thing is - everything worked like a charm until today, when
I wanted to add ops logging to rados, and restarted the gw.
After that I reverted to the previous config, but the error persists.
the gateway is reached by nginx via fastcgi:
fastcgi_pass_request_headers on;
access_log /var/log/nginx/access.log;
include fastcgi_params; # default content in this file
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param LENGTH $content_length;
fastcgi_pass unix:/var/run/ceph.radosgw.gateway.fastcgi.sock;
radosgw config in ceph.conf:
[client.radosgw.gateway]
host = cfgate01
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw print continue = false
rgw enable usage log = true
rgw enable ops log = true
rgw dns name = fs2.smcloud.net
debug rgw = 20
I've turned on debugging and can see the length and content_length
parameters in radosgw log, or maybe I'm missing something:
2015-12-17 15:20:47.934962 7f5e98f99700 20 enqueued request
req=0x7f5ef4032380
2015-12-17 15:20:47.934976 7f5e98f99700 20 RGWWQ:
2015-12-17 15:20:47.934977 7f5e98f99700 20 req: 0x7f5ef4032380
2015-12-17 15:20:47.934987 7f5e98f99700 10 allocated request
req=0x7f5ef4032bd0
2015-12-17 15:20:47.937538 7f5e94f91700 20 dequeued request
req=0x7f5ef4032380
2015-12-17 15:20:47.937555 7f5e94f91700 20 RGWWQ: empty
2015-12-17 15:20:47.937789 7f5e94f91700 20 CONTENT_LENGTH=5
2015-12-17 15:20:47.937796 7f5e94f91700 20 CONTENT_TYPE=text/plain
2015-12-17 15:20:47.937797 7f5e94f91700 20 DOCUMENT_ROOT=/etc/nginx/html
2015-12-17 15:20:47.937798 7f5e94f91700 20 DOCUMENT_URI=/PUT/test.txt
2015-12-17 15:20:47.937799 7f5e94f91700 20 FCGI_ROLE=RESPONDER
2015-12-17 15:20:47.937799 7f5e94f91700 20 GATEWAY_INTERFACE=CGI/1.1
2015-12-17 15:20:47.937800 7f5e94f91700 20 HTTP_ACCEPT_ENCODING=identity
2015-12-17 15:20:47.937800 7f5e94f91700 20 HTTP_AUTHORIZATION=AWS
LZ8YD48VSS0MCBYGG7XC:3frBFFTsiN0QhGEig+wu0aSLwzE=
2015-12-17 15:20:47.937801 7f5e94f91700 20 HTTP_CONTENT_LENGTH=5
2015-12-17 15:20:47.937802 7f5e94f91700 20 HTTP_CONTENT_TYPE=text/plain
2015-12-17 15:20:47.937802 7f5e94f91700 20 HTTP_HOST=eg-hls.fs2.smcloud.net
2015-12-17 15:20:47.937803 7f5e94f91700 20 HTTP_X_AMZ_DATE=Thu, 17 Dec
2015 14:20:48 +0000
2015-12-17 15:20:47.937803 7f5e94f91700 20
HTTP_X_AMZ_META_S3CMD_ATTRS=uid:1000/gname:jj/uname:jjarosiewicz/gid:1000/mode:33188/mtime:1450359091/atime:1450359106/md5:d8e8fca2dc0f896fd7cb4cb0031ba249/ctime:1450359091
2015-12-17 15:20:47.937805 7f5e94f91700 20 LENGTH=5
2015-12-17 15:20:47.937806 7f5e94f91700 20 QUERY_STRING=
2015-12-17 15:20:47.937807 7f5e94f91700 20 REDIRECT_STATUS=200
2015-12-17 15:20:47.937807 7f5e94f91700 20 REMOTE_ADDR=212.180.240.7
2015-12-17 15:20:47.937808 7f5e94f91700 20 REMOTE_PORT=53086
2015-12-17 15:20:47.937808 7f5e94f91700 20 REQUEST_METHOD=PUT
2015-12-17 15:20:47.937809 7f5e94f91700 20 REQUEST_URI=/test.txt
2015-12-17 15:20:47.937809 7f5e94f91700 20 SCRIPT_NAME=/PUT/test.txt
2015-12-17 15:20:47.937810 7f5e94f91700 20 SERVER_ADDR=212.180.241.218
2015-12-17 15:20:47.937810 7f5e94f91700 20 SERVER_NAME=adm-fs2.smcloud.net
2015-12-17 15:20:47.937810 7f5e94f91700 20 SERVER_PORT=80
2015-12-17 15:20:47.937811 7f5e94f91700 20 SERVER_PROTOCOL=HTTP/1.1
2015-12-17 15:20:47.937811 7f5e94f91700 20 SERVER_SOFTWARE=nginx/1.6.2
2015-12-17 15:20:47.937812 7f5e94f91700 1 ====== starting new request
req=0x7f5ef4032380 =====
2015-12-17 15:20:47.937929 7f5e94f91700 2 req 131:0.000117::PUT
/test.txt::initializing for trans_id =
tx000000000000000000083-005672c4bf-1d3068-default
2015-12-17 15:20:47.937945 7f5e94f91700 10 host=eg-hls.fs2.smcloud.net
2015-12-17 15:20:47.937949 7f5e94f91700 20 subdomain=eg-hls
domain=fs2.smcloud.net in_hosted_domain=1
2015-12-17 15:20:47.937965 7f5e94f91700 10 meta>> HTTP_X_AMZ_DATE
2015-12-17 15:20:47.937974 7f5e94f91700 10 meta>>
HTTP_X_AMZ_META_S3CMD_ATTRS
2015-12-17 15:20:47.937978 7f5e94f91700 10 x>> x-amz-date:Thu, 17 Dec
2015 14:20:48 +0000
2015-12-17 15:20:47.937980 7f5e94f91700 10 x>>
x-amz-meta-s3cmd-attrs:uid:1000/gname:jj/uname:jjarosiewicz/gid:1000/mode:33188/mtime:1450359091/atime:1450359106/md5:d8e8fca2dc0f896fd7cb4cb0031ba249/ctime:1450359091
2015-12-17 15:20:47.938078 7f5e94f91700 10 s->object=test.txt
s->bucket=eg-hls
2015-12-17 15:20:47.938090 7f5e94f91700 2 req 131:0.000278:s3:PUT
/test.txt::getting op
2015-12-17 15:20:47.938095 7f5e94f91700 2 req 131:0.000283:s3:PUT
/test.txt:put_obj:authorizing
2015-12-17 15:20:47.938326 7f5e94f91700 10 get_canon_resource():
dest=/eg-hls/test.txt
2015-12-17 15:20:47.938338 7f5e94f91700 10 auth_hdr:
PUT
text/plain
x-amz-date:Thu, 17 Dec 2015 14:20:48 +0000
x-amz-meta-s3cmd-attrs:uid:1000/gname:jj/uname:jjarosiewicz/gid:1000/mode:33188/mtime:1450359091/atime:1450359106/md5:d8e8fca2dc0f896fd7cb4cb0031ba249/ctime:1450359091
/eg-hls/test.txt
2015-12-17 15:20:47.938548 7f5e94f91700 15 calculated
digest=3frBFFTsiN0QhGEig+wu0aSLwzE=
2015-12-17 15:20:47.938558 7f5e94f91700 15
auth_sign=3frBFFTsiN0QhGEig+wu0aSLwzE=
2015-12-17 15:20:47.938559 7f5e94f91700 15 compare=0
2015-12-17 15:20:47.938564 7f5e94f91700 2 req 131:0.000752:s3:PUT
/test.txt:put_obj:reading permissions
2015-12-17 15:20:47.938904 7f5e94f91700 15 Read
AccessControlPolicy<AccessControlPolicy
xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner><ID>sm-catchup-uploader</ID><DisplayName>Konto
do wgrywania plików
CatchUP</DisplayName></Owner><AccessControlList><Grant><Grantee
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:type="CanonicalUser"><ID>sm-catchup-uploader</ID><DisplayName>Konto
do wgrywania plików
CatchUP</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2015-12-17 15:20:47.938924 7f5e94f91700 2 req 131:0.001111:s3:PUT
/test.txt:put_obj:init op
2015-12-17 15:20:47.938930 7f5e94f91700 2 req 131:0.001118:s3:PUT
/test.txt:put_obj:verifying op mask
2015-12-17 15:20:47.938934 7f5e94f91700 20 required_mask= 2 user.op_mask=7
2015-12-17 15:20:47.938935 7f5e94f91700 2 req 131:0.001123:s3:PUT
/test.txt:put_obj:verifying op permissions
2015-12-17 15:20:47.938940 7f5e94f91700 5 Searching permissions for
uid=sm-catchup-uploader mask=50
2015-12-17 15:20:47.938941 7f5e94f91700 5 Found permission: 15
2015-12-17 15:20:47.938941 7f5e94f91700 5 Searching permissions for
group=1 mask=50
2015-12-17 15:20:47.938942 7f5e94f91700 5 Permissions for group not found
2015-12-17 15:20:47.938943 7f5e94f91700 5 Searching permissions for
group=2 mask=50
2015-12-17 15:20:47.938943 7f5e94f91700 5 Permissions for group not found
2015-12-17 15:20:47.938944 7f5e94f91700 5 Getting permissions
id=sm-catchup-uploader owner=sm-catchup-uploader perm=2
2015-12-17 15:20:47.938944 7f5e94f91700 10 uid=sm-catchup-uploader
requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2015-12-17 15:20:47.938945 7f5e94f91700 2 req 131:0.001133:s3:PUT
/test.txt:put_obj:verifying op params
2015-12-17 15:20:47.938947 7f5e94f91700 2 req 131:0.001135:s3:PUT
/test.txt:put_obj:executing
2015-12-17 15:20:47.939771 7f5e94f91700 2 req 131:0.001959:s3:PUT
/test.txt:put_obj:http status=411
2015-12-17 15:20:47.939779 7f5e94f91700 1 ====== req done
req=0x7f5ef4032380 http_status=411 ======
2015-12-17 15:20:47.939787 7f5e94f91700 20 process_request() returned -2011
J
--
Jacek Jarosiewicz
Administrator Systemów Informatycznych
----------------------------------------------------------------------------------------
SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie
ul. Senatorska 13/15, 00-075 Warszawa
Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego
Rejestru Sądowego,
nr KRS 0000029537; kapitał zakładowy 42.756.000 zł
NIP: 957-05-49-503
Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa
----------------------------------------------------------------------------------------
SUPERMEDIA -> http://www.supermedia.pl
dostep do internetu - hosting - kolokacja - lacza - telefonia
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
