I was looking at the same thing myself, and Boto seems to work ok
(tested a 6G file - some sample code attached).
Regards
Mark
On 27/10/13 11:46, Derek Yarnell wrote:
Hi Shain,
Yes we have tested and have working S3 Multipart support for files >5GB
(RHEL64/0.67.4).
However, crossftp unless you have pro it would seem does not support
multipart. Dragondisk gives the error that I have seen when using a PUT
and not multipart, EntityTooLarge. My guess is that it is not doing
Multipart. Now from s3cmd's source it does support multipart have you
tried to give it a --debug 2 which will output boto's debug output I
believe. That should tell you more about if it is correctly doing a
multipart upload.
Thanks,
derek
On 10/26/13, 5:10 PM, Shain Miley wrote:
Hi,
I am wondering if anyone has successfully been able to upload files
larger than 5GB using radosgw.
I have tried using various clients, including dragondisk, crossftp,
s3cmd, etc...all of them have failed with a 'permission denied' response.
Each of the clients say they support multi-part (and it appears that
they do as files larger than 15MB gets split into multiple pieces)
however I can only upload files smaller than 5GB up to this point.
I am using 0.67.4 and Ubuntu 12.04.
Thanks in advance,
Shain
Shain Miley | Manager of Systems and Infrastructure, Digital Media |
smiley@xxxxxxx | 202.513.3649
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
#!/usr/bin/python
#
# Sample using mulitpart upload in boto.
import os
import boto
import boto.s3.connection
access_key = 'your-key-here'
secret_key = 'your-secret-here'
partsize = 1073741824
parts = 0
n = 0
conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = 'your-gateway-host',
is_secure=False, # uncommmnt if you are not using ssl
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
bucket = conn.create_bucket('bucketbig')
objname = 'big.dat'
filename = '/tmp/big.dat'
# figure out how many parts
filesize = os.path.getsize(filename)
parts = (filesize / partsize) + 1
print " begin upload of " + filename
print " size " + str(filesize) + ", " + str(parts) + " parts"
part = bucket.initiate_multipart_upload(objname)
fp = open(filename, 'r')
for n in range(1, parts + 1):
if (filesize - (n - 1) * partsize < partsize):
size = filesize - (n - 1) * partsize
else:
size = partsize
print " upload part " + str(n) + " size " + str(size)
part.upload_part_from_file(fp = fp, part_num = n, size = size)
print " end upload"
part.complete_upload()
fp.close()
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
