----- Original Message ----- > From: "Sean" <seapasulli@xxxxxxxxxxxx> > To: ceph-users@xxxxxxxxxxxxxx > Sent: Thursday, May 7, 2015 3:35:14 PM > Subject: RGW - Can't download complete object > > I have another thread goign on about truncation of objects and I believe > this is a separate but equally bad issue in civetweb/radosgw. My cluster > is completely healthy > > I have one (possibly more) objects stored in ceph rados gateway that > will return a different size every time I Try to download it:: > > http://pastebin.com/hK1iqXZH --- ceph -s > http://pastebin.com/brmxQRu3 --- radosgw-admin object stat of the object The two interesting things that I see here is: - the multipart upload size for each part is on the big side (is it 1GB for each part?) - it seems that there are a lot of parts that suffered from retries, could be a source for the 512k issue > http://pastebin.com/5TnvgMrX --- python download code > > The weird part is every time I download the file it is of a different > size. I am grabbing the individual objects of the 14g file and will > update this email once I have them all statted out. Currently I am > getting, on average, 1.5G to 2Gb files when the total object should be > 14G in size. > > lacadmin@kh10-9:~$ python corruptpull.py > the download failed. The filesize = 2125988202. The actual size is > 14577056082. Attempts = 1 > the download failed. The filesize = 2071462250. The actual size is > 14577056082. Attempts = 2 > the download failed. The filesize = 2016936298. The actual size is > 14577056082. Attempts = 3 > the download failed. The filesize = 1643643242. The actual size is > 14577056082. Attempts = 4 > the download failed. The filesize = 1597505898. The actual size is > 14577056082. Attempts = 5 > the download failed. The filesize = 2075656554. The actual size is > 14577056082. Attempts = 6 > the download failed. The filesize = 650117482. The actual size is > 14577056082. Attempts = 7 > the download failed. The filesize = 1987576170. The actual size is > 14577056082. Attempts = 8 > the download failed. The filesize = 2109210986. The actual size is > 14577056082. Attempts = 9 > the download failed. The filesize = 2142765418. The actual size is > 14577056082. Attempts = 10 > the download failed. The filesize = 2134376810. The actual size is > 14577056082. Attempts = 11 > the download failed. The filesize = 2146959722. The actual size is > 14577056082. Attempts = 12 > the download failed. The filesize = 2142765418. The actual size is > 14577056082. Attempts = 13 > the download failed. The filesize = 1467482474. The actual size is > 14577056082. Attempts = 14 > the download failed. The filesize = 2046296426. The actual size is > 14577056082. Attempts = 15 > the download failed. The filesize = 2021130602. The actual size is > 14577056082. Attempts = 16 > the download failed. The filesize = 1773666666. The actual size is > 14577056082. Attempts = 17 > the download failed. The filesize = 2146959722. The actual size is > 14577056082. Attempts = 18 > the download failed. The filesize = 2016936298. The actual size is > 14577056082. Attempts = 19 > the download failed. The filesize = 1983381866. The actual size is > 14577056082. Attempts = 20 > the download failed. The filesize = 2134376810. The actual size is > 14577056082. Attempts = 21 > > Notice it is always different. Once the rados -p .rgw.buckets ls | grep > finishes I will return the listing of objects as well but this is quite > odd and I think this is a separate issue. > > Has anyone seen this before? Why wouldn't radosgw return an error and > why am I getting different file sizes? Usually that means that there was some error in the middle of the download, maybe client to radosgw communication issue. What does the radosgw show when this happens? > > I would post the log from radosgw but I don't see any "err|wrn|fatal" > mentions in the log and the client completes without issue every time. > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |