This sounds like this bug: http://tracker.ceph.com/issues/17076 Will be fixed in 10.2.6. It's triggered by aws4 auth, so a workaround would be to use aws2 instead. Yehuda On Wed, Mar 1, 2017 at 10:46 AM, John Nielsen <lists@xxxxxxxxxxxx> wrote: > Hi all- > > We use Amazon S3 quite a bit at $WORK but are evaluating Ceph+radosgw as an alternative for some things. We have an "S3 smoke test" written using the AWS Java SDK that we use to validate a number of operations. On my Kraken cluster, multi-part uploads work fine for s3cmd. Our smoke test also passes fine using version 1.9.27 of the AWS SDK. However in SDK 1.11.69 the multi-part upload fails. The initial POST (to reserve the object name and start the upload) succeeds, but the first PUT fails with a 403 error. > > So, does anyone know offhand what might be going on here? If not, how can I get more details about the 403 error and what is causing it? > > The cluster was installed with Jewel and recently updated to Kraken. Using the built-in civetweb server. > > Here is the log output for three multi-part uploads. The first two are s3cmd and the older SDK, respectively. The last is the failing one with the newer SDK. > > S3cmd, Succeeds. > 2017-03-01 17:33:16.845613 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== > 2017-03-01 17:33:16.856522 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== > 2017-03-01 17:33:16.856628 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "POST /testdomobucket10x3x104x64250438/multipartStreamTest?uploads HTTP/1.1" 1 0 - - > 2017-03-01 17:33:16.953967 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== > 2017-03-01 17:33:24.094134 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== > 2017-03-01 17:33:24.094211 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "PUT /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=1&uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - - > 2017-03-01 17:33:24.193747 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== > 2017-03-01 17:33:30.002050 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== > 2017-03-01 17:33:30.002124 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "PUT /testdomobucket10x3x104x64250438/multipartStreamTest?partNumber=2&uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - - > 2017-03-01 17:33:30.085033 7f80b06de700 1 ====== starting new request req=0x7f80b06d8340 ===== > 2017-03-01 17:33:30.104944 7f80b06de700 1 ====== req done req=0x7f80b06d8340 op status=0 http_status=200 ====== > 2017-03-01 17:33:30.105007 7f80b06de700 1 civetweb: 0x7f81131fd000: 10.251.50.7 - - [01/Mar/2017:17:33:16 +0000] "POST /testdomobucket10x3x104x64250438/multipartStreamTest?uploadId=2~IGYuZC4uDC27TGWfpFkKk-Makqvk_XB HTTP/1.1" 1 0 - - > > AWS SDK (1.9.27). Succeeds. > 2017-03-01 17:54:50.720093 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== > 2017-03-01 17:54:50.733109 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== > 2017-03-01 17:54:50.733188 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "POST /testdomobucket10x3x104x6443285/multipartStreamTest?uploads HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 > 2017-03-01 17:54:50.831618 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== > 2017-03-01 17:54:58.057011 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== > 2017-03-01 17:54:58.057082 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "PUT /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo&partNumber=1 HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 > 2017-03-01 17:54:58.143235 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== > 2017-03-01 17:54:58.328351 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== > 2017-03-01 17:54:58.328437 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "PUT /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo&partNumber=2 HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 > 2017-03-01 17:54:58.415890 7f80c0eff700 1 ====== starting new request req=0x7f80c0ef9340 ===== > 2017-03-01 17:54:58.438199 7f80c0eff700 1 ====== req done req=0x7f80c0ef9340 op status=0 http_status=200 ====== > 2017-03-01 17:54:58.438253 7f80c0eff700 1 civetweb: 0x7f811314c000: 10.251.50.7 - - [01/Mar/2017:17:54:42 +0000] "POST /testdomobucket10x3x104x6443285/multipartStreamTest?uploadId=2%7EPlNR4meSvAvCYtvbqz8JLlSKu5_laxo HTTP/1.1" 1 0 - aws-sdk-java/1.9.27 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 > > AWS SDK (1.11.69), fails. > 2017-03-01 17:37:31.833693 7f80c6f0b700 1 ====== starting new request req=0x7f80c6f05340 ===== > 2017-03-01 17:37:31.845262 7f80c6f0b700 1 ====== req done req=0x7f80c6f05340 op status=0 http_status=200 ====== > 2017-03-01 17:37:31.845349 7f80c6f0b700 1 civetweb: 0x7f8113108000: 10.251.50.7 - - [01/Mar/2017:17:37:31 +0000] "POST /testdomobucket10x3x104x64150440/multipartStreamTest?uploads HTTP/1.1" 1 0 - aws-sdk-java/1.11.69 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 scala/2.10.4 > 2017-03-01 17:37:32.082100 7f80c6f0b700 1 ====== starting new request req=0x7f80c6f05340 ===== > 2017-03-01 17:37:32.082701 7f80c6f0b700 1 ====== req done req=0x7f80c6f05340 op status=0 http_status=403 ====== > 2017-03-01 17:37:42.432298 7f80c6f0b700 1 civetweb: 0x7f8113108000: 10.251.50.7 - - [01/Mar/2017:17:37:31 +0000] "PUT /testdomobucket10x3x104x64150440/multipartStreamTest?uploadId=2%7EGObj1tVySTxsRvJNjJDksOhxuBtiTzW&partNumber=1 HTTP/1.1" 1 0 - aws-sdk-java/1.11.69 Mac_OS_X/10.10.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.71-b01/1.7.0_71 scala/2.10.4 > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |