While I'm usually not fond of blaming the client application, this is really the swift command line tool issue. It tries to be smart by comparing the md5sum of the object's content with the object's etag, and it breaks with multipart objects. Multipart objects is calculated differently (md5sum of the md5sum of each part). I think the swift tool has a special handling for swift large objects (which are not the same as s3 multipart objects), so that's why it works in that specific use case. Yehuda On Wed, May 11, 2016 at 7:15 AM, Saverio Proto <zioproto@xxxxxxxxx> wrote: > It does not work also the way around: > > If I upload a file with the swift client with the -S options to force > swift to make multipart: > > swift upload -S 1000000 multipart 180.mp4 > > Then I am not able to read the file with S3 > > s3cmd get s3://multipart/180.mp4 > download: 's3://multipart/180.mp4' -> './180.mp4' [1 of 1] > download: 's3://multipart/180.mp4' -> './180.mp4' [1 of 1] > 38818503 of 38818503 100% in 1s 27.32 MB/s done > WARNING: MD5 signatures do not match: > computed=961f154cc78c7bf1be3b4009c29e5a68, > received=d41d8cd98f00b204e9800998ecf8427e > > Saverio > > > 2016-05-11 16:07 GMT+02:00 Saverio Proto <zioproto@xxxxxxxxx>: >> Thank you. >> >> It is exactly a problem with multipart. >> >> So I tried two clients (s3cmd and rclone). When you upload a file in >> S3 using multipart, you are not able to read anymore this object with >> the SWIFT API because the md5 check fails. >> >> Saverio >> >> >> >> 2016-05-09 12:00 GMT+02:00 Xusangdi <xu.sangdi@xxxxxxx>: >>> Hi, >>> >>> I'm not running a cluster as yours, but I don't think the issue is caused by you using 2 APIs at the same time. >>> IIRC the dash thing is append by S3 multipart upload, with a following digit indicating the number of parts. >>> You may want to check this reported in s3cmd community: >>> https://sourceforge.net/p/s3tools/bugs/123/ >>> >>> and some basic info from Amazon: >>> http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html >>> >>> Hope this helps :D >>> >>> Regards, >>> ---Sandy >>> >>>> -----Original Message----- >>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Saverio Proto >>>> Sent: Monday, May 09, 2016 4:42 PM >>>> To: ceph-users@xxxxxxxxxxxxxx >>>> Subject: Re: RadosGW - Problems running the S3 and SWIFT API at the same time >>>> >>>> I try to simplify the question to get some feedback. >>>> >>>> Is anyone running the RadosGW in production with S3 and SWIFT API active at the same time ? >>>> >>>> thank you ! >>>> >>>> Saverio >>>> >>>> >>>> 2016-05-06 11:39 GMT+02:00 Saverio Proto <zioproto@xxxxxxxxx>: >>>> > Hello, >>>> > >>>> > We have been running the Rados GW with the S3 API and we did not have >>>> > problems for more than a year. >>>> > >>>> > We recently enabled also the SWIFT API for our users. >>>> > >>>> > radosgw --version >>>> > ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) >>>> > >>>> > The idea is that each user of the system is free of choosing the S3 >>>> > client or the SWIFT client to access the same container/buckets. >>>> > >>>> > Please tell us if this is possible by design or if we are doing something wrong. >>>> > >>>> > We have now a problem that some files wrote in the past with S3, >>>> > cannot be read with the SWIFT API because the md5sum always fails. >>>> > >>>> > I am able to reproduce the bug in this way: >>>> > >>>> > We have this file googlebooks-fre-all-2gram-20120701-ts.gz and we know >>>> > the correct md5 is 1c8113d2bd21232688221ec74dccff3a You can download >>>> > the same file here: >>>> > https://www.dropbox.com/s/auq16vdv2maw4p7/googlebooks-fre-all-2gram-20 >>>> > 120701-ts.gz?dl=0 >>>> > >>>> > rclone mkdir lss3:bugreproduce >>>> > rclone copy googlebooks-fre-all-2gram-20120701-ts.gz lss3:bugreproduce >>>> > >>>> > The file is successfully uploaded. >>>> > >>>> > At this point I can succesfully download again the file: >>>> > rclone copy lss3:bugreproduce/googlebooks-fre-all-2gram-20120701-ts.gz >>>> > test.gz >>>> > >>>> > but not with swift: >>>> > >>>> > swift download googlebooks-ngrams-gz >>>> > fre/googlebooks-fre-all-2gram-20120701-ts.gz >>>> > Error downloading object >>>> > 'googlebooks-ngrams-gz/fre/googlebooks-fre-all-2gram-20120701-ts.gz': >>>> > u'Error downloading fre/googlebooks-fre-all-2gram-20120701-ts.gz: >>>> > md5sum != etag, 1c8113d2bd21232688221ec74dccff3a != >>>> > 1a209a31b4ac3eb923fac5e8d194d9d3-2' >>>> > >>>> > Also I found strange the dash character '-' at the end of the md5 that >>>> > is trying to compare. >>>> > >>>> > Of course upload a file with the swift client and redownloading the >>>> > same file just works. >>>> > >>>> > Should I open a bug for the radosgw on http://tracker.ceph.com/ ? >>>> > >>>> > thank you >>>> > >>>> > Saverio >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> ------------------------------------------------------------------------------------------------------------------------------------- >>> 本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出 >>> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 >>> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 >>> 邮件！ >>> This e-mail and its attachments contain confidential information from H3C, which is >>> intended only for the person or entity whose address is listed above. Any use of the >>> information contained herein in any way (including, but not limited to, total or partial >>> disclosure, reproduction, or dissemination) by persons other than the intended >>> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender >>> by phone or email immediately and delete it! > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |