Casey,
I am not sure if this is related, but I cannot seem to retrieve files that are 524288001 bytes (5MB + 1 byte) or 629145601 bytes (600MB + 1 byte) when using server side encryption. Without encryption, these files store and retrieve without issue. I'm sure there are other various permutations of lengths that would result in this issue as well. I still find it puzzling that after ingesting 167 million files in this manner, I am discovering this issue.
-Dan
On Mon, Mar 18, 2019 at 12:38 PM Dan Smith <dan.smith.11221122@xxxxxxxxx> wrote:
Hi Casey,Thanks for the quick response. I have just confirmed that if I set the PartSize to 500MB the file uploads correctly. I am hesitant to do this in production but I think we are on the right track. Interestingly enough, when I set the PartSize to 5242880 the file did not store correctly (it retrieved with a different hash, and not even the same different hash I saw before).Here are my details of my client. Please forgive me for not seeing how to attach this to the tracker item you mentioned in your email.I am using the AWSSDK.S3 nuget package version 3.3.31.24, with ceph version "ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)"My relevant c# code is as follows:using (var client = new AmazonS3Client(AccessKey, SecretKey, GetS3Config()))using (var transferUtility = new Amazon.S3.Transfer.TransferUtility(client)){var transferRequest = new Amazon.S3.Transfer.TransferUtilityUploadRequest{BucketName = bucketName,Key = key,InputStream = stream,ServerSideEncryptionCustomerProvidedKey = EncryptionKey,ServerSideEncryptionCustomerMethod = string.IsNullOrWhiteSpace(EncryptionKey) ? ServerSideEncryptionCustomerMethod.None : ServerSideEncryptionCustomerMethod.AES256};transferUtility.Upload(transferRequest);}I am having hard time locating the default PartSize this library uses, but I think it is 5MB as the content-length header is a bit more than that size. Perhaps this sanitized header is useful:PUT [redacted]/delete-me?partNumber=18&uploadId=2~2dt4pYGY3vfKxBb9FcbVlAnbz_z3HTV HTTP/1.1Expect: 100-continuex-amz-server-side-encryption-customer-algorithm: AES256x-amz-server-side-encryption-customer-key: [redacted]x-amz-server-side-encryption-customer-key-MD5: [redacted]User-Agent: aws-sdk-dotnet-coreclr/3.3.31.24 aws-sdk-dotnet-core/3.3.32.2 .NET_Core/4.6.27317.07 OS/Microsoft_Windows_10.0.17763 ClientAsync TransferManager/MultipartUploadCommandHost: [redacted]X-Amz-Date: [redacted]X-Amz-Decoded-Content-Length: 5242880X-Amz-Content-SHA256: STREAMING-AWS4-HMAC-SHA256-PAYLOADAuthorization: [redacted]Content-Length: 5248726Content-Type: text/plain14000;chunk-signature=[redacted][payload here]Thank you again for help on this matter. I'll point our vendor towards this bug as well.Cheers,DanOn Mon, Mar 18, 2019 at 12:05 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:Hi Dan,
We just got a similar report about SSE-C in
http://tracker.ceph.com/issues/38700 that seems to be related to
multipart uploads. Could you please add some details there about your s3
client, its multipart chunk size, and your ceph version?
On 3/18/19 2:38 PM, Dan Smith wrote:
> Hello,
>
> I have stored more than 167 million files in ceph using the S3 api.
> Out of those 167 million+ files, one file is not storing correctly.
>
> The file is 92MB in size. I have stored files much larger and much
> smaller. If I store the file WITHOUT using the Customer Provided
> 256-bit AES key using Server Side encryption, the file stores and
> retrieves just fine (SHA256 hashes match).
>
> If I store the file USING the 256-bit AES key using Server Side
> encryption, the file stores without error, however, when I retrieve
> the file and compare the hash of the file I retrieve from ceph against
> the hash of the original file, the hashes differ.
>
> If I store the file using Amazon S3, using the same AES key and their
> server side encryption the file stores are retrieves using out issue
> (hashes match).
>
> I can reproduce this issue in two different ceph environments.
> Thankfully, the file I am storing is not confidential, so I can share
> it out to anyone interested in this
> issue.(https://s3.amazonaws.com/aws-website-afewgoodmenrankedfantasyfootballcom-j5gvt/delete-me)
>
> I have opened a ticket with our vendor for support, but I am hoping
> someone might be able to give me some ideas on what might be going on
> as well.
>
> Cheers,
> Dan
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
