Thanks! Changing to V2 auth does indeed work around the issue with the newer SDK. > On Mar 1, 2017, at 12:33 PM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote: > > This sounds like this bug: > http://tracker.ceph.com/issues/17076 > > Will be fixed in 10.2.6. It's triggered by aws4 auth, so a workaround > would be to use aws2 instead. > > Yehuda > > > On Wed, Mar 1, 2017 at 10:46 AM, John Nielsen <lists@xxxxxxxxxxxx> wrote: >> Hi all- >> >> We use Amazon S3 quite a bit at $WORK but are evaluating Ceph+radosgw as an alternative for some things. We have an "S3 smoke test" written using the AWS Java SDK that we use to validate a number of operations. On my Kraken cluster, multi-part uploads work fine for s3cmd. Our smoke test also passes fine using version 1.9.27 of the AWS SDK. However in SDK 1.11.69 the multi-part upload fails. The initial POST (to reserve the object name and start the upload) succeeds, but the first PUT fails with a 403 error. >> >> So, does anyone know offhand what might be going on here? If not, how can I get more details about the 403 error and what is causing it? >> >> The cluster was installed with Jewel and recently updated to Kraken. Using the built-in civetweb server. >> >> Here is the log output for three multi-part uploads. The first two are s3cmd and the older SDK, respectively. The last is the failing one with the newer SDK. >> >> S3cmd, Succeeds. >> 2017-03-01 17:33:16.845613 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== >> 2017-03-01 17:33:16.856522 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== >> 2017-03-01 17:33:16.856628 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "POST /testdomobucket10x3x104x64250438/multipartStreamTest?uploads HTTP/1.1" 1 0 - - >> 2017-03-01 17:33:16.953967 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== >> 2017-03-01 17:33:24.094134 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== >> 2017-03-01 17:33:24.094211 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "PUT /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=1&uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - - >> 2017-03-01 17:33:24.193747 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== >> 2017-03-01 17:33:30.002050 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== >> 2017-03-01 17:33:30.002124 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "PUT /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=2&uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - - >> 2017-03-01 17:33:30.085033 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== >> 2017-03-01 17:33:30.104944 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== >> 2017-03-01 17:33:30.105007 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "POST /testdomobucket10x3x104x64250438/multipartStreamTest?uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - - >> >> AWS SDK (1.9.27). Succeeds. >> 2017-03-01 17:54:50.720093 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== >> 2017-03-01 17:54:50.733109 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== >> 2017-03-01 17:54:50.733188 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "POST /testdomobucket10x3x104x6443285/multipartStreamTest?uploads HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 >> 2017-03-01 17:54:50.831618 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== >> 2017-03-01 17:54:58.057011 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== >> 2017-03-01 17:54:58.057082 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "PUT /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo&partNumber=1 HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 >> 2017-03-01 17:54:58.143235 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== >> 2017-03-01 17:54:58.328351 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== >> 2017-03-01 17:54:58.328437 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "PUT /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo&partNumber=2 HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 >> 2017-03-01 17:54:58.415890 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== >> 2017-03-01 17:54:58.438199 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== >> 2017-03-01 17:54:58.438253 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "POST /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 >> >> AWS SDK (1.11.69), fails. >> 2017-03-01 17:37:31.833693 7f80c6f0b700 1 ====== starting new request req=0x7f80c6f05340 ===== >> 2017-03-01 17:37:31.845262 7f80c6f0b700 1 ====== req done req=0x7f80c6f05340 op status=0 http_status=200 ====== >> 2017-03-01 17:37:31.845349 7f80c6f0b700 1 civetweb: 0x7f8113108000: 10.251.50.7 - - [01/Mar/2017:17:37:31 +0000] "POST /testdomobucket10x3x104x64150440/multipartStreamTest?uploads HTTP/1.1" 1 0 - aws-sdk-java/1.11.69 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 scala/2.10.4 >> 2017-03-01 17:37:32.082100 7f80c6f0b700 1 ====== starting new request req=0x7f80c6f05340 ===== >> 2017-03-01 17:37:32.082701 7f80c6f0b700 1 ====== req done req=0x7f80c6f05340 op status=0 http_status=403 ====== >> 2017-03-01 17:37:42.432298 7f80c6f0b700 1 civetweb: 0x7f8113108000: 10.251.50.7 - - [01/Mar/2017:17:37:31 +0000] "PUT /testdomobucket10x3x104x64150440/multipartStreamTest?uploadId=2%7EGObj1tVySTxsRvJNjJDksOhxuBtiTzW&partNumber=1 HTTP/1.1" 1 0 - aws-sdk-java/1.11.69 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 scala/2.10.4 >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |