sending data in multiple segments causes s3upload (jets3t) to fail

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




Hi,

I am trying to use hadoop distcp while copying data from hdfs to s3. Hadoop distcp devides the data in to multiple chunks and sends the data parellely so that faster performance is achieved. However this is failing against ceph s3 indicating a mismatch between md5 and etag returned by s3. However this is working with aws s3

Is there a workaround for this, apart from setting
storage-service.disable-live-md5=true in jets3t.properties ?

Dont want to disable md5 checking because it will not ensure 
correctness of uploaded data.

Thank you,
Jaseer TK


_____________________________________________________________
The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux