In multipart upload of S3, the checksum algorithm is different from that in swift.
Including the following page for your convenience.
原始邮件
发件人: Saverio Proto<zioproto@xxxxxxxxx>
收件人: Xusangdi<xu.sangdi@xxxxxxx>
抄送: ceph-users@xxxxxxxxxxxxxx<ceph-users@xxxxxxxxxxxxxx>
发送时间: 2016年5月11日(周三) 22:15
主题: Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API atthe same time
It does not work also the way around: If I upload a file with the swift client with the -S options to force swift to make multipart: swift upload -S 1000000 multipart 180.mp4 Then I am not able to read the file with S3 s3cmd get s3://multipart/180.mp4 download: 's3://multipart/180.mp4' -> './180.mp4' [1 of 1] download: 's3://multipart/180.mp4' -> './180.mp4' [1 of 1] 38818503 of 38818503 100% in 1s 27.32 MB/s done WARNING: MD5 signatures do not match: computed=961f154cc78c7bf1be3b4009c29e5a68, received=d41d8cd98f00b204e9800998ecf8427e Saverio 2016-05-11 16:07 GMT+02:00 Saverio Proto <zioproto@xxxxxxxxx>: > Thank you. > > It is exactly a problem with multipart. > > So I tried two clients (s3cmd and rclone). When you upload a file in > S3 using multipart, you are not able to read anymore this object with > the SWIFT API because the md5 check fails. > > Saverio > > > > 2016-05-09 12:00 GMT+02:00 Xusangdi <xu.sangdi@xxxxxxx>: >> Hi, >> >> I'm not running a cluster as yours, but I don't think the issue is caused by you using 2 APIs at the same time. >> IIRC the dash thing is append by S3 multipart upload, with a following digit indicating the number of parts. >> You may want to check this reported in s3cmd community: >> https://sourceforge.net/p/s3tools/bugs/123/ >> >> and some basic info from Amazon: >> http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html >> >> Hope this helps :D >> >> Regards, >> ---Sandy >> >>> -----Original Message----- >>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Saverio Proto >>> Sent: Monday, May 09, 2016 4:42 PM >>> To: ceph-users@xxxxxxxxxxxxxx >>> Subject: Re: [ceph-users] RadosGW - Problems running the S3 and SWIFT API at the same time >>> >>> I try to simplify the question to get some feedback. >>> >>> Is anyone running the RadosGW in production with S3 and SWIFT API active at the same time ? >>> >>> thank you ! >>> >>> Saverio >>> >>> >>> 2016-05-06 11:39 GMT+02:00 Saverio Proto <zioproto@xxxxxxxxx>: >>> > Hello, >>> > >>> > We have been running the Rados GW with the S3 API and we did not have >>> > problems for more than a year. >>> > >>> > We recently enabled also the SWIFT API for our users. >>> > >>> > radosgw --version >>> > ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) >>> > >>> > The idea is that each user of the system is free of choosing the S3 >>> > client or the SWIFT client to access the same container/buckets. >>> > >>> > Please tell us if this is possible by design or if we are doing something wrong. >>> > >>> > We have now a problem that some files wrote in the past with S3, >>> > cannot be read with the SWIFT API because the md5sum always fails. >>> > >>> > I am able to reproduce the bug in this way: >>> > >>> > We have this file googlebooks-fre-all-2gram-20120701-ts.gz and we know >>> > the correct md5 is 1c8113d2bd21232688221ec74dccff3a You can download >>> > the same file here: >>> > https://www.dropbox.com/s/auq16vdv2maw4p7/googlebooks-fre-all-2gram-20 >>> > 120701-ts.gz?dl=0 >>> > >>> > rclone mkdir lss3:bugreproduce >>> > rclone copy googlebooks-fre-all-2gram-20120701-ts.gz lss3:bugreproduce >>> > >>> > The file is successfully uploaded. >>> > >>> > At this point I can succesfully download again the file: >>> > rclone copy lss3:bugreproduce/googlebooks-fre-all-2gram-20120701-ts.gz >>> > test.gz >>> > >>> > but not with swift: >>> > >>> > swift download googlebooks-ngrams-gz >>> > fre/googlebooks-fre-all-2gram-20120701-ts.gz >>> > Error downloading object >>> > 'googlebooks-ngrams-gz/fre/googlebooks-fre-all-2gram-20120701-ts.gz': >>> > u'Error downloading fre/googlebooks-fre-all-2gram-20120701-ts.gz: >>> > md5sum != etag, 1c8113d2bd21232688221ec74dccff3a != >>> > 1a209a31b4ac3eb923fac5e8d194d9d3-2' >>> > >>> > Also I found strange the dash character '-' at the end of the md5 that >>> > is trying to compare. >>> > >>> > Of course upload a file with the swift client and redownloading the >>> > same file just works. >>> > >>> > Should I open a bug for the radosgw on http://tracker.ceph.com/ ? >>> > >>> > thank you >>> > >>> > Saverio >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ------------------------------------------------------------------------------------------------------------------------------------- >> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出 >> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 >> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 >> 邮件！ >> This e-mail and its attachments contain confidential information from H3C, which is >> intended only for the person or entity whose address is listed above. Any use of the >> information contained herein in any way (including, but not limited to, total or partial >> disclosure, reproduction, or dissemination) by persons other than the intended >> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender >> by phone or email immediately and delete it!
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
