Hi Casey,
thank you for your reply, could you give me more details?
On Wed, Sep 30, 2020 at 3:56 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:
> On Wed, Sep 30, 2020 at 5:20 AM Eugeniy Khvastunov <khvastunov@xxxxxxxxx>
> wrote:
> >
> > Hi ceph-users,
> >
> > Looks like we faced a broken S3 multipart upload in Ceph 12.2.11 Luminous
> > (MCP 2019.2.0) during thanos setup.
> > The request to initialize the multipart fails.
> > S3cmd log:
> >
> > DEBUG: Sending request method_string='POST',
> > uri='/staging-prometheus-k8s-metrics/test4/dd-50M.rnd?uploads',
> > headers={'x-amz-meta-s3cmd-attrs': 'uid:1011/gname:
> >
> yevhen.kh/uname:yevhen.kh/gid:1011/mode:33188/mtime:1600198080/atime:1600198530/md5:730ee931c6058cad5bd4d87a6c8f3178/ctime:1600198363
> ',
> > 'content-type': 'application/octet-stream', 'Authorization': 'AWS
> > ZLMVD7JZZ8V09VVK66C3:YX5Oe+Ig0FPDbRpIYO2m1WHITOE=', 'x-amz-date': 'Tue,
> 15
> > Sep 2020 19:38:33 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0
> bytes)
> > DEBUG: Response: {'status': 400, 'headers': {'content-length': '253',
> > 'accept-ranges': 'bytes', 'server': 'nginx', 'connection': 'keep-alive',
> > 'x-amz-request-id': 'tx0000000000000001e019a-005f611839-deb80f-default',
> > 'date': 'Tue, 15 Sep 2020 19:38:37 GMT', 'content-type':
> > 'application/xml'}, 'reason': 'Bad Request', 'data': '<?xml version="1.0"
> >
> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>staging-prometheus-k8s-metrics</BucketName><RequestId>tx0000000000000001e019a-005f611839-deb80f-default</RequestId><HostId>deb80f-default-default</HostId></Error>'}
> > DEBUG: ConnMan.put(): connection put back to pool (
> > https://s3.cloud.f.in.company.com:8080#1)
> > DEBUG: S3Error: 400 (Bad Request)
> > DEBUG: HttpHeader: content-length: 253
> > DEBUG: HttpHeader: accept-ranges: bytes
> > DEBUG: HttpHeader: server: nginx
> > DEBUG: HttpHeader: connection: keep-alive
> > DEBUG: HttpHeader: x-amz-request-id:
> > tx0000000000000001e019a-005f611839-deb80f-default
> > DEBUG: HttpHeader: date: Tue, 15 Sep 2020 19:38:37 GMT
> > DEBUG: HttpHeader: content-type: application/xml
> > DEBUG: ErrorXML: Code: 'InvalidArgument'
> > DEBUG: ErrorXML: BucketName: 'staging-prometheus-k8s-metrics'
> > DEBUG: ErrorXML: RequestId:
> > 'tx0000000000000001e019a-005f611839-deb80f-default'
> > DEBUG: ErrorXML: HostId: 'deb80f-default-default'
> > ERROR: S3 error: 400 (InvalidArgument)
> >
> >
> > CEPH log:
> >
> > > 2020-09-10 08:26:26.238814 7f7482fdd700 1 -- 10.14.2.66:0/3902939317
> <==
> > > mon.0 10.14.2.66:6789/0 19 ==== pool_op_reply(tid 1290 (22) Invalid
> > > argument v22601) v1 ==== 43+0+0 (3815426539 0 0) 0x55faeb4e4a80 con
> > > 0x55faeabc8800
>
> it looks like the EINVAL error is coming from pool creation. we use
> the data_extra_pool (ie default.rgw.buckets.non-ec) to track some
> state for multipart uploads, so that's probably the one it's trying to
> create here
>
> > > 2020-09-10 08:26:26.241214 7f74607de700 2 req 484:1.595459:s3:POST
> > >
> /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:completing
> > > 2020-09-10 08:26:26.241389 7f74607de700 2 req 484:1.595632:s3:POST
> > >
> /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:op
> > > status=-22
> > > 2020-09-10 08:26:26.241414 7f74607de700 2 req 484:1.595658:s3:POST
> > >
> /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:http
> > > status=400
> > > 2020-09-10 08:26:26.241420 7f74607de700 1 ====== req done
> > > req=0x7f74607d8140 op status=-22 http_status=400 ======
> > > 2020-09-10 08:26:26.241443 7f74607de700 20 process_request() returned
> -22
> > > 2020-09-10 08:26:26.241492 7f74607de700 1 civetweb: 0x55faeaedf000:
> > > 10.14.5.203 - - [10/Sep/2020:08:26:24 +0000] "POST
> > >
> /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001?uploads=
> > > HTTP/1.1" 400 0 - MinIO (linux; amd64) minio-go/v7.0.2
> > > thanos-sidecar/0.15.0 (go1.14.2)
> > >
> >
> > The bucket exists. The issue is repeated on the newly created bucket and
> on
> > the existing one.
> >
> > If s3cmd runs with `--multipart-chunk-size-mb=` option and the value is
> > larger than the file size(for example, if the file is 50 mb, set
> > --multipart-chunk-size-mb=200 mb) then the file uploads.
> > Example:
> >
> > > s3cmd -c /root/f-s3-thanos.cfg put /root/dd-50M.rnd
> > > s3://staging-prometheus-k8s-metrics/test4/dd-50M.rnd -d
> > > --multipart-chunk-size-mb=200
> > >
> >
> > If s3cmd runs with `--disable-multipart` that works as well.
> >
> > Have you faced such problems?
> > I would be grateful for your advice.
> >
> > --
> > wbr. Eugeniy Khvastunov,
> > System administrator.
> > [FMGH-UANIC]
> > http://blog.unlimite.net
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
>
>
--
wbr. Eugeniy Khvastunov,
System administrator.
[FMGH-UANIC]
http://blog.unlimite.net
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
