Re: AWS SDK and MultiPart Problem

[Georgios Dimitrakakis <giorgis@xxxxxxxxxxxx>]

I 'd be more than happy to provide to you all the info but for some 
unknown reason my radosgw.log is empty.

This is the part that I have in ceph.conf

[client.radosgw.gateway]
host = xxx
keyring = /etc/ceph/keyring.radosgw.gateway
rgw socket path = /tmp/radosgw.sock
rgw dns name = xxx.example.com
rgw enable usage log = true
rgw usage log tick interval = 30
rgw usage log flush threshold = 1024
rgw usage max shards = 32
rgw usage max user shards = 1
log file = /var/log/ceph/radosgw.log
debug ms = 1
debug rgw = 20



but no matter what I put in there the log is empty....

$ pwd
/var/log/ceph
$ ls -l radosgw.log
-rw-r--r-- 1 root root 0 Nov 30 03:01 radosgw.log


I have already started  another thread with title "Empty Rados log"  
here in ceph-users list since December 4th but haven't heard from anyone 
yet...

If I solve this I will be able to provide you with all the data.


Regards,


George


> Ok, I've been digging a bit more. I don't have full radosgw logs for
> the issue, so if you could provide it (debug rgw = 20), it might 
> help.
> However, as it is now, I think the issue is with the way the client
> library is signing the requests. Instead of using the undecoded
> uploadId, it uses the encoded version for the signature, which 
> doesn't
> sign correctly. The same would have happened if it would have run
> against amazon S3 (just tested it).
> The two solutions that I see are to fix the client library, and/or to
> modify the character to one that does not require escaping. Sadly the
> dash character that you were using cannot be used safely in that
> context. Maybe tilde ('~') would could work.
>
> Yehuda
>
> On Fri, Dec 12, 2014 at 2:41 AM, Georgios Dimitrakakis
> <giorgis@xxxxxxxxxxxx> wrote:
> > Dear Yehuda,
> >
> > I have installed the patched version as you can see:
> >
> > $ radosgw --version
> > ceph version 0.80.7-1-gbd43759 
> > (bd43759f6e76fa827e2534fa4e61547779ee10a5)
> >
> > $ ceph --version
> > ceph version 0.80.7-1-gbd43759 
> > (bd43759f6e76fa827e2534fa4e61547779ee10a5)
> >
> > $ sudo yum info ceph-radosgw
> > Installed Packages
> > Name        : ceph-radosgw
> > Arch        : x86_64
> > Version     : 0.80.7
> > Release     : 1.gbd43759.el6
> > Size        : 3.8 M
> > Repo        : installed
> > From repo   : ceph-source
> > Summary     : Rados REST gateway
> > URL         : http://ceph.com/
> > License     : GPL-2.0
> > Description : radosgw is an S3 HTTP REST gateway for the RADOS 
> > object store.
> > It is
> >             : implemented as a FastCGI module using libfcgi, and can 
> > be used
> > in
> >             : conjunction with any FastCGI capable web server.
> >
> >
> > Unfortunately the problem on the multipart upload with aws-sdk still 
> > remains
> > the same!
> >
> >
> > Here is a part of the apache log:
> >
> >
> > "PUT
> >
> > /clients-space/test/iip7.dmg?partNumber=3&uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO
> > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> >
> > "PUT
> >
> > /clients-space/test/iip7.dmg?partNumber=1&uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO
> > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> >
> > "PUT
> >
> > /clients-space/test/iip7.dmg?partNumber=2&uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO
> > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> >
> >
> >
> > Directly modification of the binary so that the "2%2F" be changed to 
> > "2-"
> > results in success and here is the log:
> >
> >
> > "PUT
> >
> > /clients-space/test/iip7.dmg?partNumber=1&uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> >
> > "PUT
> >
> > /clients-space/test/iip7.dmg?partNumber=2&uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> >
> > "PUT
> >
> > /clients-space/test/iip7.dmg?partNumber=4&uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> >
> > "POST
> >
> > /clients-space/test/iip7.dmg?uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > HTTP/1.1" 200 302 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> >
> >
> >
> >
> > Can you think of something else??
> >
> >
> > Best regards,
> >
> >
> > George
> >
> >
> >
> >
> >
> > > OK! I will give it some time and will try again later!
> > >
> > > Thanks a lot for your help!
> > >
> > > Warmest regards,
> > >
> > > George
> > >
> > >
> > > > The branch I pushed earlier was based off recent development 
> > > > branch. I
> > > > just pushed one based off firefly (wip-10271-firefly). It will
> > > > probably take a bit to build.
> > > >
> > > > Yehuda
> > > >
> > > > On Thu, Dec 11, 2014 at 12:03 PM, Georgios Dimitrakakis
> > > > <giorgis@xxxxxxxxxxxx> wrote:
> > > > > Hi again!
> > > > >
> > > > > I have installed and enabled the development branch repositories 
> > > > > as
> > > > > described here:
> > > > >
> > > > >
> > > > >
> > > > > http://ceph.com/docs/master/install/get-packages/#add-ceph-development
> > > > >
> > > > > and when I try to update the ceph-radosgw package I get the 
> > > > > following:
> > > > >
> > > > > Installed Packages
> > > > > Name        : ceph-radosgw
> > > > > Arch        : x86_64
> > > > > Version     : 0.80.7
> > > > > Release     : 0.el6
> > > > > Size        : 3.8 M
> > > > > Repo        : installed
> > > > > From repo   : Ceph
> > > > > Summary     : Rados REST gateway
> > > > > URL         : http://ceph.com/
> > > > > License     : GPL-2.0
> > > > > Description : radosgw is an S3 HTTP REST gateway for the RADOS 
> > > > > object
> > > > > store.
> > > > > It is
> > > > >             : implemented as a FastCGI module using libfcgi, and 
> > > > > can be
> > > > > used
> > > > > in
> > > > >             : conjunction with any FastCGI capable web server.
> > > > >
> > > > > Available Packages
> > > > > Name        : ceph-radosgw
> > > > > Arch        : x86_64
> > > > > Epoch       : 1
> > > > > Version     : 0.80.5
> > > > > Release     : 9.el6
> > > > > Size        : 1.3 M
> > > > > Repo        : epel
> > > > > Summary     : Rados REST gateway
> > > > > URL         : http://ceph.com/
> > > > > License     : GPL-2.0
> > > > > Description : radosgw is an S3 HTTP REST gateway for the RADOS 
> > > > > object
> > > > > store.
> > > > > It is
> > > > >             : implemented as a FastCGI module using libfcgi, and 
> > > > > can be
> > > > > used
> > > > > in
> > > > >             : conjunction with any FastCGI capable web server.
> > > > >
> > > > >
> > > > >
> > > > > Is this normal???
> > > > >
> > > > > I am concerned because the installed version is 0.80.7 and the 
> > > > > available
> > > > > update package is 0.80.5
> > > > >
> > > > > Have I missed something?
> > > > >
> > > > > Regards,
> > > > >
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > > > Pushed a fix to wip-10271. Haven't tested it though, let me know 
> > > > > > if
> > > > > > you try it.
> > > > > >
> > > > > > Thanks,
> > > > > > Yehuda
> > > > > >
> > > > > > On Thu, Dec 11, 2014 at 8:38 AM, Yehuda Sadeh 
> > > > > > <yehuda@xxxxxxxxxx>
> > > > > > wrote:
> > > > > > > I don't think it has been fixed recently. I'm looking at it 
> > > > > > > now, and
> > > > > > > not sure why it hasn't triggered before in other areas.
> > > > > > >
> > > > > > > Yehuda
> > > > > > >
> > > > > > > On Thu, Dec 11, 2014 at 5:55 AM, Georgios Dimitrakakis
> > > > > > > <giorgis@xxxxxxxxxxxx> wrote:
> > > > > > > > This issue seems very similar to these:
> > > > > > > >
> > > > > > > > http://tracker.ceph.com/issues/8202
> > > > > > > > http://tracker.ceph.com/issues/8702
> > > > > > > >
> > > > > > > >
> > > > > > > > Would it make any difference if I try to build CEPH from 
> > > > > > > > sources?
> > > > > > > >
> > > > > > > > I mean is someone aware of it been fixed on any of the recent 
> > > > > > > > commits
> > > > > > > > and
> > > > > > > > probably hasn't passed yet to the repositories?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 08 Dec 2014 19:47:59 +0200, Georgios Dimitrakakis 
> > > > > > > > wrote:
> > > > > > > > > I 've just created issues #10271
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > > On Fri, 5 Dec 2014 09:30:45 -0800, Yehuda Sadeh wrote:
> > > > > > > > > > It looks like a bug. Can you open an issue on 
> > > > > > > > > > tracker.ceph.com,
> > > > > > > > > > describing what you see?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Yehuda
> > > > > > > > > >
> > > > > > > > > > On Fri, Dec 5, 2014 at 7:17 AM, Georgios Dimitrakakis
> > > > > > > > > > <giorgis@xxxxxxxxxxxx> wrote:
> > > > > > > > > > > It would be nice to see where and how "uploadId"
> > > > > > > > > > >
> > > > > > > > > > > is being calculated...
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > George
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > For example if I try to perform the same multipart upload 
> > > > > > > > > > > > at an
> > > > > > > > > > > > older
> > > > > > > > > > > > version ceph version 0.72.2
> > > > > > > > > > > > (a913ded2ff138aefb8cb84d347d72164099cfd60)
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I can see the upload ID in the apache log as:
> > > > > > > > > > > >
> > > > > > > > > > > > "PUT
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > /test/XXXX.dat?partNumber=25&uploadId=I3yihBFZmHx9CCqtcDjr8d-RhgfX8NW
> > > > > > > > > > > > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.0.29 linux/v0.10.33"
> > > > > > > > > > > >
> > > > > > > > > > > > but when I try the same at ceph version 0.80.7
> > > > > > > > > > > > (6c0127fcb58008793d3c8b62d925bc91963672a3)
> > > > > > > > > > > >
> > > > > > > > > > > > I get the following:
> > > > > > > > > > > >
> > > > > > > > > > > > "PUT
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > /test/XXXX.dat?partNumber=12&uploadId=2%2Ff9UgnHhdK0VCnMlpT-XA8ttia1HjK36
> > > > > > > > > > > > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.0.29 
> > > > > > > > > > > > linux/v0.10.33"
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > and my guess is that the "%2F" at the latter is the one 
> > > > > > > > > > > > that is
> > > > > > > > > > > > causing the problem and hence the 403 error.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > What do you think???
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > >
> > > > > > > > > > > > George
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am using AWS SDK JS v.2.0.29 to perform a multipart 
> > > > > > > > > > > > > upload
> > > > > > > > > > > > > into
> > > > > > > > > > > > > Radosgw with ceph version 0.80.7
> > > > > > > > > > > > > (6c0127fcb58008793d3c8b62d925bc91963672a3) and I am 
> > > > > > > > > > > > > getting a
> > > > > > > > > > > > > 403
> > > > > > > > > > > > > error.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I believe that the id which is send to all requests and 
> > > > > > > > > > > > > has been
> > > > > > > > > > > > > urlencoded by the aws-sdk-js doesn't match with the one 
> > > > > > > > > > > > > in rados
> > > > > > > > > > > > > because it's not urlencoded.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Is that the case? Can you confirm it?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Is there something I can do?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > George
> > > > > > > > > > > > >
> > > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > > ceph-users mailing list
> > > > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > ceph-users mailing list
> > > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > ceph-users mailing list
> > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > ceph-users mailing list
> > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@xxxxxxxxxxxxxx
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com