Along those lines, you might want to use something similar to the
attached to check for any failed/partial uploads that are taking up
space (note cannot be gc'd away automatically). I just got caught by this.
In fact the previous code I posted should probably use a try:... except:
block to cancel the upload if the program needs to abort - but it is
still possible to get failed uploads for other reasons, so it probably
still useful to have something to find any!
Cheers
Mark
On 28/10/13 18:04, Mark Kirkwood wrote:
I was looking at the same thing myself, and Boto seems to work ok
(tested a 6G file - some sample code attached).
Regards
Mark
On 27/10/13 11:46, Derek Yarnell wrote:
Hi Shain,
Yes we have tested and have working S3 Multipart support for files >5GB
(RHEL64/0.67.4).
However, crossftp unless you have pro it would seem does not support
multipart. Dragondisk gives the error that I have seen when using a PUT
and not multipart, EntityTooLarge. My guess is that it is not doing
Multipart. Now from s3cmd's source it does support multipart have you
tried to give it a --debug 2 which will output boto's debug output I
believe. That should tell you more about if it is correctly doing a
multipart upload.
Thanks,
derek
On 10/26/13, 5:10 PM, Shain Miley wrote:
Hi,
I am wondering if anyone has successfully been able to upload files
larger than 5GB using radosgw.
I have tried using various clients, including dragondisk, crossftp,
s3cmd, etc...all of them have failed with a 'permission denied'
response.
Each of the clients say they support multi-part (and it appears that
they do as files larger than 15MB gets split into multiple pieces)
however I can only upload files smaller than 5GB up to this point.
I am using 0.67.4 and Ubuntu 12.04.
Thanks in advance,
Shain
Shain Miley | Manager of Systems and Infrastructure, Digital Media |
smiley@xxxxxxx | 202.513.3649
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
#!/usr/bin/python
#
# Cancel failed uploads so their storage is released
#
import os
import boto
import boto.s3.connection
access_key = 'your key here'
secret_key = 'your secret here'
conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = 'your host here',
is_secure=False, # uncommmnt if you are not using ssl
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
for bucket in conn.get_all_buckets():
print " scanning bucket " + bucket.name
uploads = bucket.get_all_multipart_uploads()
print " found " + str(len(uploads)) + " incomplete multi-part uploads (cancelling)"
for u in uploads:
u.cancel_upload()
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
