Unable to download files from ceph radosgw node using openstack juno swift client.

Vivek Varghese Cherian <vivekcherian@xxxxxxxxx> · Tue, 16 Dec 2014 12:26:03 +0530

Hi,

I am integrating ceph firefly radosgw with openstack juno keystone, the operating system used on the ceph 
nodes and on the openstack node is Ubuntu 14.04.

I am able to create containers and upload files using the swift client to ceph. 

But when I try to download files, I am getting the following error,

ppmuser@ppm-dc-c3sv3-ju:~$ swift --verbose --debug -V 1 -A http://10.x.x.126/auth -U swift:swift -K <Key_here> download demo-container1 file_12345.txt

INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.x.x.126
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /auth HTTP/1.1" 204 0
DEBUG:swiftclient:REQ: curl -i http://10.x.x.126/auth -X GET
DEBUG:swiftclient:RESP STATUS: 204 No Content
DEBUG:swiftclient:RESP HEADERS: [('x-auth-token', 'AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e'), ('x-storage-token', 'AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e'), ('date', 'Tue, 16 Dec 2014 05:49:11 GMT'), ('x-storage-url', 'http://10.x.x.126/swift/v1'), ('server', 'Apache/2.4.7 (Ubuntu)'), ('content-type', 'application/json')]
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.x.x.126
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /swift/v1/demo-container1/file_12345.txt HTTP/1.1" 200 14
DEBUG:swiftclient:REQ: curl -i http://10.x.x.126/swift/v1/demo-container1/file_12345.txt -X GET -H "X-Auth-Token: AUTH_rgwtk0b00000073776966743a73776966740484a024256146dc5719915468deb431889a0ff3c6514ac8ae2388abdfbaac6d262c3e3e"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: [('content-length', '14'), ('accept-ranges', 'bytes'), ('server', 'Apache/2.4.7 (Ubuntu)'), ('last-modified', 'Mon, 15 Dec 2014 13:35:37 GMT'), ('etag', '94b371af646d4c53d8f3f8ff7be74dbb'), ('date', 'Tue, 16 Dec 2014 05:49:11 GMT'), ('x-object-meta-mtime', '1418650375.550963')]

Error downloading object 'demo-container1/file_12345.txt': 'Error downloading file_12345.txt: read_length != content_length, 0 != 14'

ppmuser@ppm-dc-c3sv3-ju:~

The apache2 error and access logs on the radosgw ceph node is as follows,

root@ppm-c240-ceph3:~# tail -f /var/log/apache2/error.log

[Tue Dec 16 01:10:20.735061 2014] [fastcgi:error] [pid 31426:tid 140231210624768] [client 10.x.x.175:55902] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
[Tue Dec 16 01:10:20.735134 2014] [fastcgi:error] [pid 31426:tid 140231210624768] [client 10.x.x.175:55902] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Tue Dec 16 01:10:20.735161 2014] [http:error] [pid 31426:tid 140231210624768] [client 10.x.x.175:55902] not unsetting Content-Length on HEAD response (rgw changes)\n
[Tue Dec 16 01:10:20.857215 2014] [:warn] [pid 31426:tid 140231202232064] FastCGI: 10.x.x.175 HEAD http://ppm-c240-ceph3.cisco.com/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 auth
[Tue Dec 16 01:10:50.887381 2014] [fastcgi:error] [pid 31426:tid 140231202232064] [client 10.x.x.175:55910] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
[Tue Dec 16 01:10:50.887454 2014] [fastcgi:error] [pid 31426:tid 140231202232064] [client 10.x.x.175:55910] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Tue Dec 16 01:10:50.887481 2014] [http:error] [pid 31426:tid 140231202232064] [client 10.x.x.175:55910] not unsetting Content-Length on HEAD response (rgw changes)\n
[Tue Dec 16 01:10:50.898281 2014] [:warn] [pid 31426:tid 140231193839360] FastCGI: 10.x.x.175 GET http://ppm-c240-ceph3.cisco.com/swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 auth
[Tue Dec 16 01:11:20.928476 2014] [fastcgi:error] [pid 31426:tid 140231193839360] [client 10.x.x.175:55911] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec)
[Tue Dec 16 01:11:20.928556 2014] [fastcgi:error] [pid 31426:tid 140231193839360] [client 10.x.x.175:55911] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"

root@ppm-c240-ceph3:~# tail -f /var/log/apache2/access.log

10.81.83.175 - - [16/Dec/2014:01:00:20 -0500] "HEAD /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:00:50 -0500] "GET /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500 719 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET /auth HTTP/1.1" 204 431 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET /swift/v1/demo-container1?format=json HTTP/1.1" 200 411 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:22 -0500] "GET /swift/v1/demo-container1?format=json&marker=file_12345.txt HTTP/1.1" 200 174 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:08:50 -0500] "HEAD /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:09:20 -0500] "GET /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500 719 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:09:50 -0500] "HEAD /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:10:20 -0500] "HEAD /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93 HTTP/1.1" 500 189 "-" "python-swiftclient-2.3.0"
10.81.83.175 - - [16/Dec/2014:01:10:50 -0500] "GET /swift/v1/AUTH_25bb0caaff834efdafa1c1fcbb6aaf93?format=json HTTP/1.1" 500 719 "-" "python-swiftclient-2.3.0"

The radosgw logs is as follows,

root@ppm-c240-ceph3:/var/log/radosgw# tail -f ceph-client.admin

2014-12-09 11:39:19.862854 7fdbfa1257c0  0 librados: client.admin initialization error (2) No such file or directory
2014-12-09 11:39:19.863429 7fdbfa1257c0 -1 Couldn't init storage provider (RADOS)
2014-12-09 11:41:25.894934 7ffa8fe087c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 5843
2014-12-09 11:41:25.922673 7ffa8fe087c0 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2014-12-09 11:41:25.922807 7ffa8fe087c0  0 librados: client.admin initialization error (2) No such file or directory
2014-12-09 11:41:25.924901 7ffa8fe087c0 -1 Couldn't init storage provider (RADOS)
2014-12-09 11:42:12.110334 7f145f77d7c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 5862
2014-12-09 11:42:12.139115 7f145f77d7c0 -1 monclient(hunting): ERROR: missing keyring, cannot use cephx for authentication
2014-12-09 11:42:12.139215 7f145f77d7c0  0 librados: client.admin initialization error (2) No such file or directory
2014-12-09 11:42:12.141559 7f145f77d7c0 -1 Couldn't init storage provider (RADOS)

Any pointers as to what could be the cause of the error in downloading files from the ceph cluster using the openstack juno swift client would be highly appreciated.

Regards,
--- 
Vivek Varghese Cherian 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com