Re: S3 Multi-part upload broken with newer AWS Java SDK and Kraken RGW

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hey Yehuda,

Are there plans to port of this fix to Kraken?  (or is there even another Kraken release planned? :)

thanks!

-Ben

On Wed, Mar 1, 2017 at 11:33 AM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:
This sounds like this bug:
http://tracker.ceph.com/issues/17076

Will be fixed in 10.2.6. It's triggered by aws4 auth, so a workaround
would be to use aws2 instead.

Yehuda


On Wed, Mar 1, 2017 at 10:46 AM, John Nielsen <lists@xxxxxxxxxxxx> wrote:
> Hi all-
>
> We use Amazon S3 quite a bit at $WORK but are evaluating Ceph+radosgw as an alternative for some things. We have an "S3 smoke test" written using the AWS Java SDK that we use to validate a number of operations. On my Kraken cluster, multi-part uploads work fine for s3cmd. Our smoke test also passes fine using version 1.9.27 of the AWS SDK. However in SDK 1.11.69 the multi-part upload fails. The initial POST (to reserve the object name and start the upload) succeeds, but the first PUT fails with a 403 error.
>
> So, does anyone know offhand what might be going on here? If not, how can I get more details about the 403 error and what is causing it?
>
> The cluster was installed with Jewel and recently updated to Kraken. Using the built-in civetweb server.
>
> Here is the log output for three multi-part uploads. The first two are s3cmd and the older SDK, respectively. The last is the failing one with the newer SDK.
>
> S3cmd, Succeeds.
> 2017-03-01 17:33:16.845613 7f80b06de700  1 ====== starting new request req=0x7f80b06d8340 =====
> 2017-03-01 17:33:16.856522 7f80b06de700  1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ======
> 2017-03-01 17:33:16.856628 7f80b06de700  1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "POST /testdomobucket10x3x104x64250438/multipartStreamTest?uploads HTTP/1.1" 1 0 - -
> 2017-03-01 17:33:16.953967 7f80b06de700  1 ====== starting new request req=0x7f80b06d8340 =====
> 2017-03-01 17:33:24.094134 7f80b06de700  1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ======
> 2017-03-01 17:33:24.094211 7f80b06de700  1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "PUT /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=1&uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - -
> 2017-03-01 17:33:24.193747 7f80b06de700  1 ====== starting new request req=0x7f80b06d8340 =====
> 2017-03-01 17:33:30.002050 7f80b06de700  1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ======
> 2017-03-01 17:33:30.002124 7f80b06de700  1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "PUT /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=2&uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - -
> 2017-03-01 17:33:30.085033 7f80b06de700  1 ====== starting new request req=0x7f80b06d8340 =====
> 2017-03-01 17:33:30.104944 7f80b06de700  1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ======
> 2017-03-01 17:33:30.105007 7f80b06de700  1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "POST /testdomobucket10x3x104x64250438/multipartStreamTest?uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - -
>
> AWS SDK (1.9.27). Succeeds.
> 2017-03-01 17:54:50.720093 7f80c0eff700  1 ====== starting new request req=0x7f80c0ef9340 =====
> 2017-03-01 17:54:50.733109 7f80c0eff700  1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ======
> 2017-03-01 17:54:50.733188 7f80c0eff700  1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "POST /testdomobucket10x3x104x6443285/multipartStreamTest?uploads HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71
> 2017-03-01 17:54:50.831618 7f80c0eff700  1 ====== starting new request req=0x7f80c0ef9340 =====
> 2017-03-01 17:54:58.057011 7f80c0eff700  1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ======
> 2017-03-01 17:54:58.057082 7f80c0eff700  1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "PUT /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo&partNumber=1 HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71
> 2017-03-01 17:54:58.143235 7f80c0eff700  1 ====== starting new request req=0x7f80c0ef9340 =====
> 2017-03-01 17:54:58.328351 7f80c0eff700  1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ======
> 2017-03-01 17:54:58.328437 7f80c0eff700  1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "PUT /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo&partNumber=2 HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71
> 2017-03-01 17:54:58.415890 7f80c0eff700  1 ====== starting new request req=0x7f80c0ef9340 =====
> 2017-03-01 17:54:58.438199 7f80c0eff700  1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ======
> 2017-03-01 17:54:58.438253 7f80c0eff700  1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "POST /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71
>
> AWS SDK (1.11.69), fails.
> 2017-03-01 17:37:31.833693 7f80c6f0b700  1 ====== starting new request req=0x7f80c6f05340 =====
> 2017-03-01 17:37:31.845262 7f80c6f0b700  1 ====== req done req=0x7f80c6f05340 op status=0 http_status=200 ======
> 2017-03-01 17:37:31.845349 7f80c6f0b700  1 civetweb: 0x7f8113108000: 10.251.50.7 - - [01/Mar/2017:17:37:31 +0000] "POST /testdomobucket10x3x104x64150440/multipartStreamTest?uploads HTTP/1.1" 1 0 - aws-sdk-java/1.11.69 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 scala/2.10.4
> 2017-03-01 17:37:32.082100 7f80c6f0b700  1 ====== starting new request req=0x7f80c6f05340 =====
> 2017-03-01 17:37:32.082701 7f80c6f0b700  1 ====== req done req=0x7f80c6f05340 op status=0 http_status=403 ======
> 2017-03-01 17:37:42.432298 7f80c6f0b700  1 civetweb: 0x7f8113108000: 10.251.50.7 - - [01/Mar/2017:17:37:31 +0000] "PUT /testdomobucket10x3x104x64150440/multipartStreamTest?uploadId=2%7EGObj1tVySTxsRvJNjJDksOhxuBtiTzW&partNumber=1 HTTP/1.1" 1 0 - aws-sdk-java/1.11.69 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 scala/2.10.4
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux