CloudStack 4.2 - radosgw / S3 storage issues

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hello guys,

I am doing a test ACS setup to see how we can use Ceph for both Primary and Secondary storage services. I have now successfully added both Primary (cluster wide) and Secondary storage. However, I've noticed that my SSVM and CPVM are not being created, so digging in the logs revealed the following exceptions:



The radosgw logs show the following:

2013-10-29 00:19:38.289487 7f2aa7d9f780 20 enqueued request req=0x2390060
2013-10-29 00:19:38.289518 7f2aa7d9f780 20 RGWWQ:
2013-10-29 00:19:38.289521 7f2aa7d9f780 20 req: 0x2390060
2013-10-29 00:19:38.289529 7f2aa7d9f780 10 allocated request req=0x23452f0
2013-10-29 00:19:38.289572 7f2aa7d9f780 20 enqueued request req=0x23452f0
2013-10-29 00:19:38.289575 7f2aa7d9f780 20 RGWWQ:
2013-10-29 00:19:38.289576 7f2aa7d9f780 20 req: 0x2390060
2013-10-29 00:19:38.289578 7f2aa7d9f780 20 req: 0x23452f0
2013-10-29 00:19:38.289610 7f2aa7d9f780 10 allocated request req=0x23a1630
2013-10-29 00:19:38.289613 7f2a54ff9700 20 dequeued request req=0x2390060
2013-10-29 00:19:38.289627 7f2a54ff9700 20 RGWWQ:
2013-10-29 00:19:38.289629 7f2a54ff9700 20 req: 0x23452f0
2013-10-29 00:19:38.289647 7f2a54ff9700  1 ====== starting new request req=0x2390060 =====
2013-10-29 00:19:38.289650 7f2a36fcd700 20 dequeued request req=0x23452f0
2013-10-29 00:19:38.289675 7f2a36fcd700 20 RGWWQ: empty
2013-10-29 00:19:38.289685 7f2a36fcd700  1 ====== starting new request req=0x23452f0 =====
2013-10-29 00:19:38.289715 7f2a54ff9700  2 req 1291:0.000069::POST /template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2::initializing
2013-10-29 00:19:38.289723 7f2a54ff9700 10 host=cloudstack-secondary.arh-ibstorage1domain-name.com rgw_dns_name=arh-ibstorage1-ibdomain-name.com
2013-10-29 00:19:38.289755 7f2a36fcd700  2 req 1292:0.000069::POST /template%2Ftmpl%2F1%2F3%2Frouting-3%2Fsystemvmtemplate-2013-06-12-master-kvm.qcow2.bz2::initializing
2013-10-29 00:19:38.289761 7f2a36fcd700 10 host=cloudstack-secondary.arh-ibstorage1domain-name.com rgw_dns_name=arh-ibstorage1-ibdomain-name.com
2013-10-29 00:19:38.289761 7f2a54ff9700 10 s->object=tmpl/1/4/centos55-x86_64/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2 s->bucket=template
2013-10-29 00:19:38.289770 7f2a54ff9700 20 FCGI_ROLE=RESPONDER
2013-10-29 00:19:38.289771 7f2a54ff9700 20 SCRIPT_URL=/template/tmpl/1/4/centos55-x86_64/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2
2013-10-29 00:19:38.289773 7f2a54ff9700 20 SCRIPT_URI=http://cloudstack-secondary.arh-ibstorage1domain-name.com/template/tmpl/1/4/centos55-x86_64/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.b
z2
2013-10-29 00:19:38.289775 7f2a54ff9700 20 HTTP_AUTHORIZATION=AWS S3-User-Key:v1NjAqxoFbROJOlBPRWyOSw8IZI=
2013-10-29 00:19:38.289776 7f2a54ff9700 20 HTTP_HOST=cloudstack-secondary.arh-ibstorage1domain-name.com
2013-10-29 00:19:38.289776 7f2a54ff9700 20 HTTP_DATE=Tue, 29 Oct 2013 00:19:38 GMT
2013-10-29 00:19:38.289777 7f2a54ff9700 20 HTTP_USER_AGENT=aws-sdk-java/1.3.22 Linux/3.5.0-42-generic OpenJDK_64-Bit_Server_VM/20.0-b12
2013-10-29 00:19:38.289778 7f2a54ff9700 20 CONTENT_TYPE=application/x-bzip2
2013-10-29 00:19:38.289780 7f2a54ff9700 20 HTTP_TRANSFER_ENCODING=chunked
2013-10-29 00:19:38.289782 7f2a54ff9700 20 HTTP_CONNECTION=Keep-Alive
2013-10-29 00:19:38.289784 7f2a54ff9700 20 PATH=/usr/local/bin:/usr/bin:/bin
2013-10-29 00:19:38.289785 7f2a54ff9700 20 SERVER_SIGNATURE=
2013-10-29 00:19:38.289786 7f2a54ff9700 20 SERVER_SOFTWARE=Apache/2.2.22 (Ubuntu)
2013-10-29 00:19:38.289787 7f2a54ff9700 20 SERVER_NAME=cloudstack-secondary.arh-ibstorage1domain-name.com
2013-10-29 00:19:38.289788 7f2a54ff9700 20 SERVER_ADDR=192.168.169.200
2013-10-29 00:19:38.289789 7f2a54ff9700 20 SERVER_PORT=80
2013-10-29 00:19:38.289790 7f2a54ff9700 20 REMOTE_ADDR=192.168.169.1
2013-10-29 00:19:38.289790 7f2a36fcd700 10 s->object=tmpl/1/3/routing-3/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2 s->bucket=template
2013-10-29 00:19:38.289791 7f2a54ff9700 20 DOCUMENT_ROOT=/var/www
2013-10-29 00:19:38.289794 7f2a54ff9700 20 SCRIPT_FILENAME=/var/www/s3gw.fcgi
2013-10-29 00:19:38.289794 7f2a54ff9700 20 REMOTE_PORT=34613
2013-10-29 00:19:38.289795 7f2a54ff9700 20 GATEWAY_INTERFACE=CGI/1.1
2013-10-29 00:19:38.289796 7f2a54ff9700 20 SERVER_PROTOCOL=HTTP/1.1
2013-10-29 00:19:38.289797 7f2a54ff9700 20 REQUEST_METHOD=POST
2013-10-29 00:19:38.289796 7f2a36fcd700 20 FCGI_ROLE=RESPONDER
2013-10-29 00:19:38.289798 7f2a54ff9700 20 QUERY_STRING=page=template&params=/tmpl/1/4/centos55-x86_64/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2&uploads
2013-10-29 00:19:38.289798 7f2a36fcd700 20 SCRIPT_URL=/template/tmpl/1/3/routing-3/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2
2013-10-29 00:19:38.289799 7f2a54ff9700 20 REQUEST_URI=/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2?uploads
2013-10-29 00:19:38.289800 7f2a36fcd700 20 SCRIPT_URI=http://cloudstack-secondary.arh-ibstorage1domain-name.com/template/tmpl/1/3/routing-3/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2
2013-10-29 00:19:38.289800 7f2a54ff9700 20 SCRIPT_NAME=/template/tmpl/1/4/centos55-x86_64/eec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2
2013-10-29 00:19:38.289801 7f2a36fcd700 20 HTTP_AUTHORIZATION=AWS S3-User-Key:ycqus/8tCxjtrI4B6VNndllruZI=
2013-10-29 00:19:38.289802 7f2a36fcd700 20 HTTP_HOST=cloudstack-secondary.arh-ibstorage1domain-name.com
2013-10-29 00:19:38.289803 7f2a36fcd700 20 HTTP_DATE=Tue, 29 Oct 2013 00:19:38 GMT
2013-10-29 00:19:38.289802 7f2a54ff9700  2 req 1291:0.000156:s3:POST /template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2::getting op
2013-10-29 00:19:38.289805 7f2a36fcd700 20 HTTP_USER_AGENT=aws-sdk-java/1.3.22 Linux/3.5.0-42-generic OpenJDK_64-Bit_Server_VM/20.0-b12
2013-10-29 00:19:38.289806 7f2a36fcd700 20 CONTENT_TYPE=application/octet-stream
2013-10-29 00:19:38.289807 7f2a36fcd700 20 HTTP_TRANSFER_ENCODING=chunked
2013-10-29 00:19:38.289808 7f2a36fcd700 20 HTTP_CONNECTION=Keep-Alive
2013-10-29 00:19:38.289809 7f2a36fcd700 20 PATH=/usr/local/bin:/usr/bin:/bin
2013-10-29 00:19:38.289810 7f2a36fcd700 20 SERVER_SIGNATURE=
2013-10-29 00:19:38.289808 7f2a54ff9700  2 req 1291:0.000162:s3:POST /template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2:init_multipart:authorizing
2013-10-29 00:19:38.289811 7f2a36fcd700 20 SERVER_SOFTWARE=Apache/2.2.22 (Ubuntu)
2013-10-29 00:19:38.289812 7f2a36fcd700 20 SERVER_NAME=cloudstack-secondary.arh-ibstorage1domain-name.com
2013-10-29 00:19:38.289813 7f2a36fcd700 20 SERVER_ADDR=192.168.169.200
2013-10-29 00:19:38.289814 7f2a36fcd700 20 SERVER_PORT=80
2013-10-29 00:19:38.289815 7f2a36fcd700 20 REMOTE_ADDR=192.168.169.1
2013-10-29 00:19:38.289816 7f2a36fcd700 20 DOCUMENT_ROOT=/var/www
2013-10-29 00:19:38.289818 7f2a36fcd700 20 SCRIPT_FILENAME=/var/www/s3gw.fcgi
2013-10-29 00:19:38.289819 7f2a36fcd700 20 REMOTE_PORT=34612
2013-10-29 00:19:38.289820 7f2a36fcd700 20 GATEWAY_INTERFACE=CGI/1.1
2013-10-29 00:19:38.289820 7f2a36fcd700 20 SERVER_PROTOCOL=HTTP/1.1
2013-10-29 00:19:38.289821 7f2a36fcd700 20 REQUEST_METHOD=POST
2013-10-29 00:19:38.289822 7f2a36fcd700 20 QUERY_STRING=page=template&params=/tmpl/1/3/routing-3/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2&uploads
2013-10-29 00:19:38.289823 7f2a36fcd700 20 REQUEST_URI=/template%2Ftmpl%2F1%2F3%2Frouting-3%2Fsystemvmtemplate-2013-06-12-master-kvm.qcow2.bz2?uploads
2013-10-29 00:19:38.289824 7f2a36fcd700 20 SCRIPT_NAME=/template/tmpl/1/3/routing-3/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2
2013-10-29 00:19:38.289825 7f2a36fcd700  2 req 1292:0.000140:s3:POST /template%2Ftmpl%2F1%2F3%2Frouting-3%2Fsystemvmtemplate-2013-06-12-master-kvm.qcow2.bz2::getting op
2013-10-29 00:19:38.289830 7f2a36fcd700  2 req 1292:0.000145:s3:POST /template%2Ftmpl%2F1%2F3%2Frouting-3%2Fsystemvmtemplate-2013-06-12-master-kvm.qcow2.bz2:init_multipart:authorizing
2013-10-29 00:19:38.289834 7f2a54ff9700 20 get_obj_state: rctx=0x7f2a0000af50 obj=.users:S3-User-Key state=0x7f2a00009598 s->prefetch_data=0
2013-10-29 00:19:38.289844 7f2a54ff9700 10 moving .users+S3-User-Key to cache LRU end
2013-10-29 00:19:38.289847 7f2a54ff9700 10 cache get: name=.users+S3-User-Key : hit
2013-10-29 00:19:38.289845 7f2a36fcd700 20 get_obj_state: rctx=0x7f292c003130 obj=.users:S3-User-Key state=0x7f292c10c168 s->prefetch_data=0
2013-10-29 00:19:38.289853 7f2a36fcd700 10 moving .users+S3-User-Key to cache LRU end
2013-10-29 00:19:38.289855 7f2a36fcd700 10 cache get: name=.users+S3-User-Key : hit
2013-10-29 00:19:38.289856 7f2a54ff9700 20 get_obj_state: s->obj_tag was set empty
2013-10-29 00:19:38.289861 7f2a36fcd700 20 get_obj_state: s->obj_tag was set empty
2013-10-29 00:19:38.289862 7f2a54ff9700 10 moving .users+S3-User-Key to cache LRU end
2013-10-29 00:19:38.289864 7f2a54ff9700 10 cache get: name=.users+S3-User-Key : hit
2013-10-29 00:19:38.289866 7f2a36fcd700 10 moving .users+S3-User-Key to cache LRU end
2013-10-29 00:19:38.289868 7f2a36fcd700 10 cache get: name=.users+S3-User-Key : hit
2013-10-29 00:19:38.289882 7f2a54ff9700 20 get_obj_state: rctx=0x7f2a00009e50 obj=.users.uid:cloudstack state=0x7f2a00009c58 s->prefetch_data=0
2013-10-29 00:19:38.289884 7f2a36fcd700 20 get_obj_state: rctx=0x7f292c003aa0 obj=.users.uid:cloudstack state=0x7f292c10e3f8 s->prefetch_data=0
2013-10-29 00:19:38.289888 7f2a54ff9700 10 moving .users.uid+cloudstack to cache LRU end
2013-10-29 00:19:38.289890 7f2a54ff9700 10 cache get: name=.users.uid+cloudstack : hit
2013-10-29 00:19:38.289905 7f2a54ff9700 20 get_obj_state: s->obj_tag was set empty
2013-10-29 00:19:38.289906 7f2a36fcd700 10 moving .users.uid+cloudstack to cache LRU end
2013-10-29 00:19:38.289909 7f2a36fcd700 10 cache get: name=.users.uid+cloudstack : hit
2013-10-29 00:19:38.289926 7f2a36fcd700 20 get_obj_state: s->obj_tag was set empty
2013-10-29 00:19:38.289926 7f2a54ff9700 10 moving .users.uid+cloudstack to cache LRU end
2013-10-29 00:19:38.289929 7f2a54ff9700 10 cache get: name=.users.uid+cloudstack : hit
2013-10-29 00:19:38.289931 7f2a36fcd700 10 moving .users.uid+cloudstack to cache LRU end
2013-10-29 00:19:38.289933 7f2a36fcd700 10 cache get: name=.users.uid+cloudstack : hit
2013-10-29 00:19:38.289980 7f2a54ff9700 10 get_canon_resource(): dest=
2013-10-29 00:19:38.289980 7f2a36fcd700 10 get_canon_resource(): dest=
2013-10-29 00:19:38.289985 7f2a36fcd700 10 auth_hdr:
POST

application/octet-stream
Tue, 29 Oct 2013 00:19:38 GMT
/template%2Ftmpl%2F1%2F3%2Frouting-3%2Fsystemvmtemplate-2013-06-12-master-kvm.qcow2.bz2?uploads
2013-10-29 00:19:38.289985 7f2a54ff9700 10 auth_hdr:
POST

application/x-bzip2
Tue, 29 Oct 2013 00:19:38 GMT
/template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2?uploads
2013-10-29 00:19:38.290060 7f2a54ff9700 15 calculated digest=Hb7LF7RPK0wmBQeqUzud21nx5pQ=
2013-10-29 00:19:38.290060 7f2a36fcd700 15 calculated digest=mJw/9jormcA8g3HxX9YLpv25uHQ=
2013-10-29 00:19:38.290063 7f2a54ff9700 15 auth_sign=v1NjAqxoFbROJOlBPRWyOSw8IZI=
2013-10-29 00:19:38.290063 7f2a36fcd700 15 auth_sign=ycqus/8tCxjtrI4B6VNndllruZI=
2013-10-29 00:19:38.290064 7f2a54ff9700 15 compare=46
2013-10-29 00:19:38.290065 7f2a36fcd700 15 compare=12
2013-10-29 00:19:38.290067 7f2a54ff9700 10 failed to authorize request
2013-10-29 00:19:38.290068 7f2a36fcd700 10 failed to authorize request
2013-10-29 00:19:38.290097 7f2a54ff9700  2 req 1291:0.000451:s3:POST /template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2:init_multipart:http status=403
2013-10-29 00:19:38.290097 7f2a36fcd700  2 req 1292:0.000412:s3:POST /template%2Ftmpl%2F1%2F3%2Frouting-3%2Fsystemvmtemplate-2013-06-12-master-kvm.qcow2.bz2:init_multipart:http status=403
2013-10-29 00:19:38.290231 7f2a36fcd700  1 ====== req done req=0x23452f0 http_status=403 ======
2013-10-29 00:19:38.290433 7f2a54ff9700  1 ====== req done req=0x2390060 http_status=403 ======


The apache log file shows 403
192.168.169.1 - - [29/Oct/2013:10:44:13 +0000] "POST /template%2Ftmpl%2F1%2F4%2Fcentos55-x86_64%2Feec2209b-9875-3c8d-92be-c001bd8a0faf.qcow2.bz2?uploads HTTP/1.1" 403 302 "-" "aws-sdk-java/1.3.22 Linux/3.5.0-42-generic OpenJDK_64-Bit_Server_VM/20.0-b12"


From the above I can see that there seems to be some issues with access, which is strange as I can successfully run rest-bench tool using the same credentials and the same bucket. Does anyone know what I am missing here?

Many thanks

Andrei
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux