On Wed, Sep 30, 2020 at 5:20 AM Eugeniy Khvastunov <khvastunov@xxxxxxxxx> wrote:
>
> Hi ceph-users,
>
> Looks like we faced a broken S3 multipart upload in Ceph 12.2.11 Luminous
> (MCP 2019.2.0) during thanos setup.
> The request to initialize the multipart fails.
> S3cmd log:
>
> DEBUG: Sending request method_string='POST',
> uri='/staging-prometheus-k8s-metrics/test4/dd-50M.rnd?uploads',
> headers={'x-amz-meta-s3cmd-attrs': 'uid:1011/gname:
> yevhen.kh/uname:yevhen.kh/gid:1011/mode:33188/mtime:1600198080/atime:1600198530/md5:730ee931c6058cad5bd4d87a6c8f3178/ctime:1600198363',
> 'content-type': 'application/octet-stream', 'Authorization': 'AWS
> ZLMVD7JZZ8V09VVK66C3:YX5Oe+Ig0FPDbRpIYO2m1WHITOE=', 'x-amz-date': 'Tue, 15
> Sep 2020 19:38:33 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0 bytes)
> DEBUG: Response: {'status': 400, 'headers': {'content-length': '253',
> 'accept-ranges': 'bytes', 'server': 'nginx', 'connection': 'keep-alive',
> 'x-amz-request-id': 'tx0000000000000001e019a-005f611839-deb80f-default',
> 'date': 'Tue, 15 Sep 2020 19:38:37 GMT', 'content-type':
> 'application/xml'}, 'reason': 'Bad Request', 'data': '<?xml version="1.0"
> encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>staging-prometheus-k8s-metrics</BucketName><RequestId>tx0000000000000001e019a-005f611839-deb80f-default</RequestId><HostId>deb80f-default-default</HostId></Error>'}
> DEBUG: ConnMan.put(): connection put back to pool (
> https://s3.cloud.f.in.company.com:8080#1)
> DEBUG: S3Error: 400 (Bad Request)
> DEBUG: HttpHeader: content-length: 253
> DEBUG: HttpHeader: accept-ranges: bytes
> DEBUG: HttpHeader: server: nginx
> DEBUG: HttpHeader: connection: keep-alive
> DEBUG: HttpHeader: x-amz-request-id:
> tx0000000000000001e019a-005f611839-deb80f-default
> DEBUG: HttpHeader: date: Tue, 15 Sep 2020 19:38:37 GMT
> DEBUG: HttpHeader: content-type: application/xml
> DEBUG: ErrorXML: Code: 'InvalidArgument'
> DEBUG: ErrorXML: BucketName: 'staging-prometheus-k8s-metrics'
> DEBUG: ErrorXML: RequestId:
> 'tx0000000000000001e019a-005f611839-deb80f-default'
> DEBUG: ErrorXML: HostId: 'deb80f-default-default'
> ERROR: S3 error: 400 (InvalidArgument)
>
>
> CEPH log:
>
> > 2020-09-10 08:26:26.238814 7f7482fdd700 1 -- 10.14.2.66:0/3902939317 <==
> > mon.0 10.14.2.66:6789/0 19 ==== pool_op_reply(tid 1290 (22) Invalid
> > argument v22601) v1 ==== 43+0+0 (3815426539 0 0) 0x55faeb4e4a80 con
> > 0x55faeabc8800
it looks like the EINVAL error is coming from pool creation. we use
the data_extra_pool (ie default.rgw.buckets.non-ec) to track some
state for multipart uploads, so that's probably the one it's trying to
create here
> > 2020-09-10 08:26:26.241214 7f74607de700 2 req 484:1.595459:s3:POST
> > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:completing
> > 2020-09-10 08:26:26.241389 7f74607de700 2 req 484:1.595632:s3:POST
> > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:op
> > status=-22
> > 2020-09-10 08:26:26.241414 7f74607de700 2 req 484:1.595658:s3:POST
> > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:http
> > status=400
> > 2020-09-10 08:26:26.241420 7f74607de700 1 ====== req done
> > req=0x7f74607d8140 op status=-22 http_status=400 ======
> > 2020-09-10 08:26:26.241443 7f74607de700 20 process_request() returned -22
> > 2020-09-10 08:26:26.241492 7f74607de700 1 civetweb: 0x55faeaedf000:
> > 10.14.5.203 - - [10/Sep/2020:08:26:24 +0000] "POST
> > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001?uploads=
> > HTTP/1.1" 400 0 - MinIO (linux; amd64) minio-go/v7.0.2
> > thanos-sidecar/0.15.0 (go1.14.2)
> >
>
> The bucket exists. The issue is repeated on the newly created bucket and on
> the existing one.
>
> If s3cmd runs with `--multipart-chunk-size-mb=` option and the value is
> larger than the file size(for example, if the file is 50 mb, set
> --multipart-chunk-size-mb=200 mb) then the file uploads.
> Example:
>
> > s3cmd -c /root/f-s3-thanos.cfg put /root/dd-50M.rnd
> > s3://staging-prometheus-k8s-metrics/test4/dd-50M.rnd -d
> > --multipart-chunk-size-mb=200
> >
>
> If s3cmd runs with `--disable-multipart` that works as well.
>
> Have you faced such problems?
> I would be grateful for your advice.
>
> --
> wbr. Eugeniy Khvastunov,
> System administrator.
> [FMGH-UANIC]
> http://blog.unlimite.net
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
