Hi Casey,
Thanks for the quick response. I have just confirmed that if I set the PartSize to 500MB the file uploads correctly. I am hesitant to do this in production but I think we are on the right track. Interestingly enough, when I set the PartSize to 5242880 the file did not store correctly (it retrieved with a different hash, and not even the same different hash I saw before).
Here are my details of my client. Please forgive me for not seeing how to attach this to the tracker item you mentioned in your email.
I am using the AWSSDK.S3 nuget package version 3.3.31.24, with ceph version "ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)"
My relevant c# code is as follows:
using (var client = new AmazonS3Client(AccessKey, SecretKey, GetS3Config()))
using (var transferUtility = new Amazon.S3.Transfer.TransferUtility(client))
{
var transferRequest = new Amazon.S3.Transfer.TransferUtilityUploadRequest
{
BucketName = bucketName,
Key = key,
InputStream = stream,
ServerSideEncryptionCustomerProvidedKey = EncryptionKey,
ServerSideEncryptionCustomerMethod = string.IsNullOrWhiteSpace(EncryptionKey) ? ServerSideEncryptionCustomerMethod.None : ServerSideEncryptionCustomerMethod.AES256
};
transferUtility.Upload(transferRequest);
}
I am having hard time locating the default PartSize this library uses, but I think it is 5MB as the content-length header is a bit more than that size. Perhaps this sanitized header is useful:
PUT [redacted]/delete-me?partNumber=18&uploadId=2~2dt4pYGY3vfKxBb9FcbVlAnbz_z3HTV HTTP/1.1
Expect: 100-continue
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: [redacted]
x-amz-server-side-encryption-customer-key-MD5: [redacted]
User-Agent: aws-sdk-dotnet-coreclr/3.3.31.24 aws-sdk-dotnet-core/3.3.32.2 .NET_Core/4.6.27317.07 OS/Microsoft_Windows_10.0.17763 ClientAsync TransferManager/MultipartUploadCommand
Host: [redacted]
X-Amz-Date: [redacted]
X-Amz-Decoded-Content-Length: 5242880
X-Amz-Content-SHA256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
Authorization: [redacted]
Content-Length: 5248726
Content-Type: text/plain
14000;chunk-signature=[redacted]
[payload here]
Thank you again for help on this matter. I'll point our vendor towards this bug as well.
Cheers,
Dan
On Mon, Mar 18, 2019 at 12:05 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:
Hi Dan,
We just got a similar report about SSE-C in
http://tracker.ceph.com/issues/38700 that seems to be related to
multipart uploads. Could you please add some details there about your s3
client, its multipart chunk size, and your ceph version?
On 3/18/19 2:38 PM, Dan Smith wrote:
> Hello,
>
> I have stored more than 167 million files in ceph using the S3 api.
> Out of those 167 million+ files, one file is not storing correctly.
>
> The file is 92MB in size. I have stored files much larger and much
> smaller. If I store the file WITHOUT using the Customer Provided
> 256-bit AES key using Server Side encryption, the file stores and
> retrieves just fine (SHA256 hashes match).
>
> If I store the file USING the 256-bit AES key using Server Side
> encryption, the file stores without error, however, when I retrieve
> the file and compare the hash of the file I retrieve from ceph against
> the hash of the original file, the hashes differ.
>
> If I store the file using Amazon S3, using the same AES key and their
> server side encryption the file stores are retrieves using out issue
> (hashes match).
>
> I can reproduce this issue in two different ceph environments.
> Thankfully, the file I am storing is not confidential, so I can share
> it out to anyone interested in this
> issue.(https://s3.amazonaws.com/aws-website-afewgoodmenrankedfantasyfootballcom-j5gvt/delete-me)
>
> I have opened a ticket with our vendor for support, but I am hoping
> someone might be able to give me some ideas on what might be going on
> as well.
>
> Cheers,
> Dan
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
