Yep,
I was blindly adding one extra part to cope with the effects of integer
division. I guess I could cast to floats etc and then take the ceil of
the result, but just checking if the file size is an exact multiple of
the part size seems the least error prone and clearest (diff attached
for your enjoyment).
Cheers
Mark
On 01/11/13 09:51, Mark Kirkwood wrote:
Blast -I must have some shoddy arithmetic around the bit where I work
out the final pirce size. I'll experient...
Cheers
Mark
On 01/11/13 06:35, Shain Miley wrote:
PS...I tested the cancel script it worked like a charm (the script
found quite a few partial files that needed to be removed).
Thanks again,
Shain
Shain Miley | Manager of Systems and Infrastructure, Digital Media |
smiley@xxxxxxx | 202.513.3649
________________________________________
From: ceph-users-bounces@xxxxxxxxxxxxxx
[ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of Shain Miley
[SMiley@xxxxxxx]
Sent: Thursday, October 31, 2013 1:27 PM
To: Mark Kirkwood; derek@xxxxxxxxxxxxxx; ceph-users@xxxxxxxx
Subject: Re: Radosgw and large files
Mark,
Thanks for the update.
Just an FYI I ran into an issue using the script when it turned out
that the last part of the file was exactly 0 bytes. in length.
For example:
begin upload of root.img
size 10737418240, 11 parts
upload part 1 size 1073741824
upload part 2 size 1073741824
upload part 3 size 1073741824
upload part 4 size 1073741824
upload part 5 size 1073741824
upload part 6 size 1073741824
upload part 7 size 1073741824
upload part 8 size 1073741824
upload part 9 size 1073741824
upload part 10 size 1073741824
upload part 11 size 0
Traceback (most recent call last):
File "/home/smiley/Downloads/s3-big.py", line 44, in <module>
part.upload_part_from_file(fp = fp, part_num = n, size = size)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/multipart.py",
line 240, in upload_part_from_file
query_args=query_args, size=size)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line
946, in set_contents_from_file
'fp is at EOF. Use rewind option or seek() to data start.')
AttributeError: fp is at EOF. Use rewind option or seek() to data start.
I was testing the following file:
-rw-r--r-- 1 root root 10737418240 Oct 29 10:36 root.img
Thanks again,
Shain
Shain Miley | Manager of Systems and Infrastructure, Digital Media |
smiley@xxxxxxx | 202.513.3649
________________________________________
From: Mark Kirkwood [mark.kirkwood@xxxxxxxxxxxxxxx]
Sent: Wednesday, October 30, 2013 8:29 PM
To: derek@xxxxxxxxxxxxxx; Shain Miley; ceph-users@xxxxxxxx
Subject: Re: Radosgw and large files
Along those lines, you might want to use something similar to the
attached to check for any failed/partial uploads that are taking up
space (note cannot be gc'd away automatically). I just got caught by
this.
In fact the previous code I posted should probably use a try:... except:
block to cancel the upload if the program needs to abort - but it is
still possible to get failed uploads for other reasons, so it probably
still useful to have something to find any!
Cheers
Mark
On 28/10/13 18:04, Mark Kirkwood wrote:
I was looking at the same thing myself, and Boto seems to work ok
(tested a 6G file - some sample code attached).
Regards
Mark
On 27/10/13 11:46, Derek Yarnell wrote:
Hi Shain,
Yes we have tested and have working S3 Multipart support for files >5GB
(RHEL64/0.67.4).
However, crossftp unless you have pro it would seem does not support
multipart. Dragondisk gives the error that I have seen when using a
PUT
and not multipart, EntityTooLarge. My guess is that it is not doing
Multipart. Now from s3cmd's source it does support multipart have you
tried to give it a --debug 2 which will output boto's debug output I
believe. That should tell you more about if it is correctly doing a
multipart upload.
Thanks,
derek
On 10/26/13, 5:10 PM, Shain Miley wrote:
Hi,
I am wondering if anyone has successfully been able to upload files
larger than 5GB using radosgw.
I have tried using various clients, including dragondisk, crossftp,
s3cmd, etc...all of them have failed with a 'permission denied'
response.
Each of the clients say they support multi-part (and it appears that
they do as files larger than 15MB gets split into multiple pieces)
however I can only upload files smaller than 5GB up to this point.
I am using 0.67.4 and Ubuntu 12.04.
Thanks in advance,
Shain
Shain Miley | Manager of Systems and Infrastructure, Digital Media |
smiley@xxxxxxx | 202.513.3649
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
*** s3-big.py.orig 2013-11-01 11:32:57.673748111 +1300
--- s3-big.py 2013-11-01 11:32:08.313746868 +1300
***************
*** 29,35 ****
# figure out how many parts
filesize = os.path.getsize(filename)
! parts = (filesize / partsize) + 1
print " begin upload of " + filename
print " size " + str(filesize) + ", " + str(parts) + " parts"
--- 29,38 ----
# figure out how many parts
filesize = os.path.getsize(filename)
! if filesize % partsize == 0:
! parts = (filesize / partsize)
! else:
! parts = (filesize / partsize) + 1
print " begin upload of " + filename
print " size " + str(filesize) + ", " + str(parts) + " parts"
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
