radosgw: Multipart upload fails while using AWS Java SDK and signature v4

Shaon <mdshaonimran@xxxxxxxxx> · Mon, 20 Mar 2017 09:55:15 -0700

Hi,

When I try to do a multipart upload using s3cmd and signature v4 against ceph (ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)) it succeeds [1].

But when I try to do the same thing with AWS Java SDK (1.11.97), it fails with 403 SignatureDoesNotMatch.

(FYI, it works fine with signature version 2 and upload without mpu with sigv4 works as well.)

In both cases I am trying to upload a 17mb file with 15mb part size.

SignatureDoesNotMatch exception from AWS Java SDK:

2017-03-20 09:29:09.933 DEBUG wire:72 - http-outgoing-0 >> "POST /testbucket/testkeyawssdk?uploads HTTP/1.1[\r][\n]"
2017-03-20 09:29:09.936 DEBUG wire:72 - http-outgoing-0 >> "Host: 10.111.5.141:7480[\r][\n]"
2017-03-20 09:29:09.937 DEBUG wire:72 - http-outgoing-0 >> "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
2017-03-20 09:29:09.937 DEBUG wire:72 - http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=4J31KSQ9040IGL2DDA7Y/20170320/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=c4a36f357f8d87601dfd7b1cb5262988c3594d568bfb5a756bc4ad5fa09ac5c0[\r][\n]"
2017-03-20 09:29:09.937 DEBUG wire:72 - http-outgoing-0 >> "X-Amz-Date: 20170320T162904Z[\r][\n]"
2017-03-20 09:29:09.937 DEBUG wire:72 - http-outgoing-0 >> "User-Agent: aws-sdk-java/1.11.97 Mac_OS_X/10.12.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.66-b17/1.8.0_66[\r][\n]"
2017-03-20 09:29:09.938 DEBUG wire:72 - http-outgoing-0 >> "amz-sdk-invocation-id: 404a6785-b78b-e30d-7748-59c06ae5ae97[\r][\n]"
2017-03-20 09:29:09.938 DEBUG wire:72 - http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2017-03-20 09:29:09.938 DEBUG wire:72 - http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
2017-03-20 09:29:09.938 DEBUG wire:72 - http-outgoing-0 >> "Content-Length: 0[\r][\n]"
2017-03-20 09:29:09.938 DEBUG wire:72 - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2017-03-20 09:29:09.938 DEBUG wire:72 - http-outgoing-0 >> "[\r][\n]"
2017-03-20 09:29:09.997 DEBUG wire:72 - http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
2017-03-20 09:29:09.998 DEBUG wire:72 - http-outgoing-0 << "x-amz-request-id: tx000000000000000002010-0058d0034e-d83b-default[\r][\n]"
2017-03-20 09:29:09.998 DEBUG wire:72 - http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
2017-03-20 09:29:09.999 DEBUG wire:72 - http-outgoing-0 << "Content-Length: 254[\r][\n]"
2017-03-20 09:29:10.001 DEBUG wire:72 - http-outgoing-0 << "Date: Mon, 20 Mar 2017 16:29:02 GMT[\r][\n]"
2017-03-20 09:29:10.001 DEBUG wire:72 - http-outgoing-0 << "Connection: Keep-Alive[\r][\n]"
2017-03-20 09:29:10.001 DEBUG wire:72 - http-outgoing-0 << "[\r][\n]"
2017-03-20 09:29:10.046 DEBUG wire:86 - http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>testbucket</Bucket><Key>testkeyawssdk</Key><UploadId>2~o3Ixqoi-90fZeQCVYpwAJloD5S3iz5S</UploadId></InitiateMultipartUploadResult>"
2017-03-20 09:29:10.059 DEBUG wire:72 - http-outgoing-0 >> "PUT /testbucket/testkeyawssdk?uploadId=2%7Eo3Ixqoi-90fZeQCVYpwAJloD5S3iz5S&partNumber=1 HTTP/1.1[\r][\n]"
2017-03-20 09:29:10.059 DEBUG wire:72 - http-outgoing-0 >> "Host: 10.111.5.141:7480[\r][\n]"
2017-03-20 09:29:10.059 DEBUG wire:72 - http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
2017-03-20 09:29:10.060 DEBUG wire:72 - http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=4J31KSQ9040IGL2DDA7Y/20170320/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=63a15ad0819c4a309f0772dd41273eb91303d29b29e296bd3be178e86ef71a32[\r][\n]"
2017-03-20 09:29:10.060 DEBUG wire:72 - http-outgoing-0 >> "X-Amz-Date: 20170320T162910Z[\r][\n]"
2017-03-20 09:29:10.060 DEBUG wire:72 - http-outgoing-0 >> "User-Agent: aws-sdk-java/1.11.97 Mac_OS_X/10.12.3 Java_HotSpot(TM)_64-Bit_Server_VM/25.66-b17/1.8.0_66[\r][\n]"
2017-03-20 09:29:10.060 DEBUG wire:72 - http-outgoing-0 >> "amz-sdk-invocation-id: fecabaaf-d753-3a2b-6b33-73bb46a71413[\r][\n]"
2017-03-20 09:29:10.060 DEBUG wire:72 - http-outgoing-0 >> "x-amz-decoded-content-length: 15728640[\r][\n]"
2017-03-20 09:29:10.060 DEBUG wire:72 - http-outgoing-0 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2017-03-20 09:29:10.060 DEBUG wire:72 - http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
2017-03-20 09:29:10.061 DEBUG wire:72 - http-outgoing-0 >> "Content-Length: 15739526[\r][\n]"
2017-03-20 09:29:10.061 DEBUG wire:72 - http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2017-03-20 09:29:10.061 DEBUG wire:72 - http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
2017-03-20 09:29:10.061 DEBUG wire:72 - http-outgoing-0 >> "[\r][\n]"
2017-03-20 09:29:10.104 DEBUG wire:72 - http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
2017-03-20 09:29:10.105 DEBUG wire:72 - http-outgoing-0 << "x-amz-request-id: tx000000000000000002011-0058d0034e-d83b-default[\r][\n]"
2017-03-20 09:29:10.106 DEBUG wire:72 - http-outgoing-0 << "Content-Length: 194[\r][\n]"
2017-03-20 09:29:10.106 DEBUG wire:72 - http-outgoing-0 << "Accept-Ranges: bytes[\r][\n]"
2017-03-20 09:29:10.106 DEBUG wire:72 - http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
2017-03-20 09:29:10.107 DEBUG wire:72 - http-outgoing-0 << "Date: Mon, 20 Mar 2017 16:29:02 GMT[\r][\n]"
2017-03-20 09:29:10.107 DEBUG wire:72 - http-outgoing-0 << "Connection: Keep-Alive[\r][\n]"
2017-03-20 09:29:10.107 DEBUG wire:72 - http-outgoing-0 << "[\r][\n]"
2017-03-20 09:29:10.146 DEBUG wire:86 - http-outgoing-0 << "<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000002011-0058d0034e-d83b-default</RequestId><HostId>d83b-default-default</HostId></Error>"

Please let me know if more information is required.

My initial assumption was that I am hitting this bug,
http://tracker.ceph.com/issues/17076

But according to the following email this was supposed to be fixed in 10.2.6 which is the version I am using.
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-March/016617.html

I was guessing if this has anything to with the way UploadId is passed in clients like s3cmd,
resource[uri]=/tempfile17mb?partNumber=1&uploadId=2~hHwYZeWx0sfY0fzQpHT2WiYw8tNp4gN

vs. clients like AWS Java SDK,
"PUT /testbucket/testkeyawssdk?uploadId=2%7Eo3Ixqoi90fZeQCVYpwAJloD5S3iz5S&partNumber=1

But then again, AWS Java SDK does the same thing while uploading using signature v2.

Please let me know if this is an issue that is likely to be fixed or there is any thing that I could do to workaround this error.

Thanks,
Shaon

[1] https://gist.github.com/shaon/c6df5d62197c638abc2b02df9d3646c6

-- 
Imran Hossain Shaon | http://shaon.me/

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com