Re: AWS SDK and MultiPart Problem

[Georgios Dimitrakakis <giorgis@xxxxxxxxxxxx>]

How silly of me!!!

I 've just noticed that the file isn't writable by the apache!


I 'll be back with the logs...


G.


> I 'd be more than happy to provide to you all the info but for some
> unknown reason my radosgw.log is empty.
>
> This is the part that I have in ceph.conf
>
> [client.radosgw.gateway]
> host = xxx
> keyring = /etc/ceph/keyring.radosgw.gateway
> rgw socket path = /tmp/radosgw.sock
> rgw dns name = xxx.example.com
> rgw enable usage log = true
> rgw usage log tick interval = 30
> rgw usage log flush threshold = 1024
> rgw usage max shards = 32
> rgw usage max user shards = 1
> log file = /var/log/ceph/radosgw.log
> debug ms = 1
> debug rgw = 20
>
>
>
> but no matter what I put in there the log is empty....
>
> $ pwd
> /var/log/ceph
> $ ls -l radosgw.log
> -rw-r--r-- 1 root root 0 Nov 30 03:01 radosgw.log
>
>
> I have already started  another thread with title "Empty Rados log"
> here in ceph-users list since December 4th but haven't heard from
> anyone yet...
>
> If I solve this I will be able to provide you with all the data.
>
>
> Regards,
>
>
> George
>
>
> > Ok, I've been digging a bit more. I don't have full radosgw logs for
> > the issue, so if you could provide it (debug rgw = 20), it might 
> > help.
> > However, as it is now, I think the issue is with the way the client
> > library is signing the requests. Instead of using the undecoded
> > uploadId, it uses the encoded version for the signature, which 
> > doesn't
> > sign correctly. The same would have happened if it would have run
> > against amazon S3 (just tested it).
> > The two solutions that I see are to fix the client library, and/or 
> > to
> > modify the character to one that does not require escaping. Sadly 
> > the
> > dash character that you were using cannot be used safely in that
> > context. Maybe tilde ('~') would could work.
> >
> > Yehuda
> >
> > On Fri, Dec 12, 2014 at 2:41 AM, Georgios Dimitrakakis
> > <giorgis@xxxxxxxxxxxx> wrote:
> > > Dear Yehuda,
> > >
> > > I have installed the patched version as you can see:
> > >
> > > $ radosgw --version
> > > ceph version 0.80.7-1-gbd43759 
> > > (bd43759f6e76fa827e2534fa4e61547779ee10a5)
> > >
> > > $ ceph --version
> > > ceph version 0.80.7-1-gbd43759 
> > > (bd43759f6e76fa827e2534fa4e61547779ee10a5)
> > >
> > > $ sudo yum info ceph-radosgw
> > > Installed Packages
> > > Name        : ceph-radosgw
> > > Arch        : x86_64
> > > Version     : 0.80.7
> > > Release     : 1.gbd43759.el6
> > > Size        : 3.8 M
> > > Repo        : installed
> > > From repo   : ceph-source
> > > Summary     : Rados REST gateway
> > > URL         : http://ceph.com/
> > > License     : GPL-2.0
> > > Description : radosgw is an S3 HTTP REST gateway for the RADOS 
> > > object store.
> > > It is
> > >             : implemented as a FastCGI module using libfcgi, and 
> > > can be used
> > > in
> > >             : conjunction with any FastCGI capable web server.
> > >
> > >
> > > Unfortunately the problem on the multipart upload with aws-sdk 
> > > still remains
> > > the same!
> > >
> > >
> > > Here is a part of the apache log:
> > >
> > >
> > > "PUT
> > >
> > > /clients-space/test/iip7.dmg?partNumber=3&uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO
> > > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> > >
> > > "PUT
> > >
> > > /clients-space/test/iip7.dmg?partNumber=1&uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO
> > > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> > >
> > > "PUT
> > >
> > > /clients-space/test/iip7.dmg?partNumber=2&uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO
> > > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> > >
> > >
> > >
> > > Directly modification of the binary so that the "2%2F" be changed 
> > > to "2-"
> > > results in success and here is the log:
> > >
> > >
> > > "PUT
> > >
> > > /clients-space/test/iip7.dmg?partNumber=1&uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> > >
> > > "PUT
> > >
> > > /clients-space/test/iip7.dmg?partNumber=2&uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> > >
> > > "PUT
> > >
> > > /clients-space/test/iip7.dmg?partNumber=4&uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> > >
> > > "POST
> > >
> > > /clients-space/test/iip7.dmg?uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2
> > > HTTP/1.1" 200 302 "-" "aws-sdk-nodejs/2.1.0 darwin/v0.10.33"
> > >
> > >
> > >
> > >
> > > Can you think of something else??
> > >
> > >
> > > Best regards,
> > >
> > >
> > > George
> > >
> > >
> > >
> > >
> > >
> > > > OK! I will give it some time and will try again later!
> > > >
> > > > Thanks a lot for your help!
> > > >
> > > > Warmest regards,
> > > >
> > > > George
> > > >
> > > >
> > > > > The branch I pushed earlier was based off recent development 
> > > > > branch. I
> > > > > just pushed one based off firefly (wip-10271-firefly). It will
> > > > > probably take a bit to build.
> > > > >
> > > > > Yehuda
> > > > >
> > > > > On Thu, Dec 11, 2014 at 12:03 PM, Georgios Dimitrakakis
> > > > > <giorgis@xxxxxxxxxxxx> wrote:
> > > > > > Hi again!
> > > > > >
> > > > > > I have installed and enabled the development branch repositories 
> > > > > > as
> > > > > > described here:
> > > > > >
> > > > > >
> > > > > >
> > > > > > http://ceph.com/docs/master/install/get-packages/#add-ceph-development
> > > > > >
> > > > > > and when I try to update the ceph-radosgw package I get the 
> > > > > > following:
> > > > > >
> > > > > > Installed Packages
> > > > > > Name        : ceph-radosgw
> > > > > > Arch        : x86_64
> > > > > > Version     : 0.80.7
> > > > > > Release     : 0.el6
> > > > > > Size        : 3.8 M
> > > > > > Repo        : installed
> > > > > > From repo   : Ceph
> > > > > > Summary     : Rados REST gateway
> > > > > > URL         : http://ceph.com/
> > > > > > License     : GPL-2.0
> > > > > > Description : radosgw is an S3 HTTP REST gateway for the RADOS 
> > > > > > object
> > > > > > store.
> > > > > > It is
> > > > > >             : implemented as a FastCGI module using libfcgi, and 
> > > > > > can be
> > > > > > used
> > > > > > in
> > > > > >             : conjunction with any FastCGI capable web server.
> > > > > >
> > > > > > Available Packages
> > > > > > Name        : ceph-radosgw
> > > > > > Arch        : x86_64
> > > > > > Epoch       : 1
> > > > > > Version     : 0.80.5
> > > > > > Release     : 9.el6
> > > > > > Size        : 1.3 M
> > > > > > Repo        : epel
> > > > > > Summary     : Rados REST gateway
> > > > > > URL         : http://ceph.com/
> > > > > > License     : GPL-2.0
> > > > > > Description : radosgw is an S3 HTTP REST gateway for the RADOS 
> > > > > > object
> > > > > > store.
> > > > > > It is
> > > > > >             : implemented as a FastCGI module using libfcgi, and 
> > > > > > can be
> > > > > > used
> > > > > > in
> > > > > >             : conjunction with any FastCGI capable web server.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Is this normal???
> > > > > >
> > > > > > I am concerned because the installed version is 0.80.7 and the 
> > > > > > available
> > > > > > update package is 0.80.5
> > > > > >
> > > > > > Have I missed something?
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Pushed a fix to wip-10271. Haven't tested it though, let me 
> > > > > > > know if
> > > > > > > you try it.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yehuda
> > > > > > >
> > > > > > > On Thu, Dec 11, 2014 at 8:38 AM, Yehuda Sadeh 
> > > > > > > <yehuda@xxxxxxxxxx>
> > > > > > > wrote:
> > > > > > > > I don't think it has been fixed recently. I'm looking at it 
> > > > > > > > now, and
> > > > > > > > not sure why it hasn't triggered before in other areas.
> > > > > > > >
> > > > > > > > Yehuda
> > > > > > > >
> > > > > > > > On Thu, Dec 11, 2014 at 5:55 AM, Georgios Dimitrakakis
> > > > > > > > <giorgis@xxxxxxxxxxxx> wrote:
> > > > > > > > > This issue seems very similar to these:
> > > > > > > > >
> > > > > > > > > http://tracker.ceph.com/issues/8202
> > > > > > > > > http://tracker.ceph.com/issues/8702
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Would it make any difference if I try to build CEPH from 
> > > > > > > > > sources?
> > > > > > > > >
> > > > > > > > > I mean is someone aware of it been fixed on any of the recent 
> > > > > > > > > commits
> > > > > > > > > and
> > > > > > > > > probably hasn't passed yet to the repositories?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > George
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, 08 Dec 2014 19:47:59 +0200, Georgios Dimitrakakis 
> > > > > > > > > wrote:
> > > > > > > > > > I 've just created issues #10271
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > George
> > > > > > > > > >
> > > > > > > > > > On Fri, 5 Dec 2014 09:30:45 -0800, Yehuda Sadeh wrote:
> > > > > > > > > > > It looks like a bug. Can you open an issue on 
> > > > > > > > > > > tracker.ceph.com,
> > > > > > > > > > > describing what you see?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Yehuda
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Dec 5, 2014 at 7:17 AM, Georgios Dimitrakakis
> > > > > > > > > > > <giorgis@xxxxxxxxxxxx> wrote:
> > > > > > > > > > > > It would be nice to see where and how "uploadId"
> > > > > > > > > > > >
> > > > > > > > > > > > is being calculated...
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > George
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > For example if I try to perform the same multipart upload 
> > > > > > > > > > > > > at an
> > > > > > > > > > > > > older
> > > > > > > > > > > > > version ceph version 0.72.2
> > > > > > > > > > > > > (a913ded2ff138aefb8cb84d347d72164099cfd60)
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I can see the upload ID in the apache log as:
> > > > > > > > > > > > >
> > > > > > > > > > > > > "PUT
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > /test/XXXX.dat?partNumber=25&uploadId=I3yihBFZmHx9CCqtcDjr8d-RhgfX8NW
> > > > > > > > > > > > > HTTP/1.1" 200 - "-" "aws-sdk-nodejs/2.0.29 
> > > > > > > > > > > > > linux/v0.10.33"
> > > > > > > > > > > > >
> > > > > > > > > > > > > but when I try the same at ceph version 0.80.7
> > > > > > > > > > > > > (6c0127fcb58008793d3c8b62d925bc91963672a3)
> > > > > > > > > > > > >
> > > > > > > > > > > > > I get the following:
> > > > > > > > > > > > >
> > > > > > > > > > > > > "PUT
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > /test/XXXX.dat?partNumber=12&uploadId=2%2Ff9UgnHhdK0VCnMlpT-XA8ttia1HjK36
> > > > > > > > > > > > > HTTP/1.1" 403 78 "-" "aws-sdk-nodejs/2.0.29 
> > > > > > > > > > > > > linux/v0.10.33"
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > and my guess is that the "%2F" at the latter is the one 
> > > > > > > > > > > > > that is
> > > > > > > > > > > > > causing the problem and hence the 403 error.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > What do you think???
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > >
> > > > > > > > > > > > > George
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am using AWS SDK JS v.2.0.29 to perform a multipart 
> > > > > > > > > > > > > > upload
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > Radosgw with ceph version 0.80.7
> > > > > > > > > > > > > > (6c0127fcb58008793d3c8b62d925bc91963672a3) and I am 
> > > > > > > > > > > > > > getting a
> > > > > > > > > > > > > > 403
> > > > > > > > > > > > > > error.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe that the id which is send to all requests and 
> > > > > > > > > > > > > > has been
> > > > > > > > > > > > > > urlencoded by the aws-sdk-js doesn't match with the one 
> > > > > > > > > > > > > > in rados
> > > > > > > > > > > > > > because it's not urlencoded.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Is that the case? Can you confirm it?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Is there something I can do?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > George
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > > > ceph-users mailing list
> > > > > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > > ceph-users mailing list
> > > > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > ceph-users mailing list
> > > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > _______________________________________________
> > > > > > > > > > ceph-users mailing list
> > > > > > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users@xxxxxxxxxxxxxx
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@xxxxxxxxxxxxxx
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com