Hi,
i was able to track down the issue further:
* Upload had been a multipart upload - Metadata are correct
* After that, the object had been copied (PUT with x-amz-copy-source) - ETag got broken
"s3cmd du" or "s3cmd ls" fails with the copied object:
[...]
DEBUG: Sending request method_string='GET', uri='/?prefix=XXXXXXXXXXX-ZZZ%2Fdata%2FL%2FKKKKK', headers={'x-amz-date': '20191219T133746Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=XXXXXXXXXXX/20191219/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=[ommited]', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (http://appliance-12f587e9ea79f037733f378276712a39.dcs.dunkel.de#1)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http:'
b'//s3.amazonaws.com/doc/2006-03-01/"><Name>[ommitedbucketname........'
b'...............]</Name><Prefix>XXXXXXXXXXX-ZZZ/data/L/KKKKK</Prefix>'
b'<Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTrunca'
b'ted><Contents><Key>XXXXXXXXXXX-ZZZ/data/L/KKKKK</Key><LastModified>2'
b'019-12-18T22:46:50.889Z</LastModified><ETag>"ae6992f728ff1763f6'
b'b2425acf6811a5-19"</ETag><Size>296557676</Size><StorageCl'
b'ass>STANDARD</StorageClass><Owner><ID>[ommited]</ID><DisplayName>[..'
b'username.]</DisplayName></Owner><Type>Normal</Type></Contents></List'
b'BucketResult>',
'headers': {'content-type': 'application/xml',
'date': 'Thu, 19 Dec 2019 13:37:46 GMT',
'transfer-encoding': 'chunked',
'x-amz-request-id': 'tx0000000000000000463e6-005dfb7d2a-1c798e3c-default'},
'reason': 'OK',
'status': 200}
ERROR: Error parsing xml: reference to invalid character number: line 1, column 377
Surprisingly, when i fetch metadata of both objects, i don`t see any pecularities:
root@cephosd01:~# radosgw-admin object stat --bucket=[ommitted] --object="XXXXXXXXXX-YYY/data/L/KKKKK"
{
"name": "XXXXXXXXXX-YYY/data/L/KKKKK",
[..]
"etag": "ae6992f728ff1763f6b2425acf6811a5-19",
"tag": "default.477728316.187337",
"manifest": {
"objs": [],
"obj_size": 296557676,
"explicit_objs": "false",
"head_size": 0,
"max_head_size": 0,
"prefix": "XXXXXXXXXX-YYY/data/L/KKKKK.2~8sxiSL8gadZHI9VHRJiRcxo5SYMWGcV",
broken:
radosgw-admin object stat --bucket=appliance-12f587e9ea79f037733f378276712a39 --object="XXXXXXXXXXX-ZZZ/data/L/KKKKK"
[..]
"etag": "ae6992f728ff1763f6b2425acf6811a5-19",
"tag": "default.477200610.3445398",
"manifest": {
"objs": [],
"obj_size": 296557676,
"explicit_objs": "false",
"head_size": 0,
"max_head_size": 0,
"prefix": "XXXXXXXXXXX-ZZZ/data/L/KKKKK.2~8sxiSL8gadZHI9VHRJiRcxo5SYMWGcV",
[..]
root@cephosd01:~# rados -p rgw.buckets getxattr "default.375689600.58_XXXXXXXXXXX-YYY/data/L/KKKKK" user.rgw.etag > /tmp/etag.broken
root@cephosd01:~# rados -p rgw.buckets getxattr "default.375689600.58_XXXXXXXXXXX-ZZZ/data/L/KKKKK" user.rgw.etag > /tmp/etag.ok
root@cephosd01:~# diff /tmp/etag.*
root@cephosd01:~# cat /tmp/etag.broken
ae6992f728ff1763f6b2425acf6811a5-19root@cephosd01:~#
I may read and write or even copy (s3cmd copy) the object again, and all is fine.
Still not clear, at what point the issue appears.
Any clues?
kind regards,
Ingo
----- Ursprüngliche Mail -----
Von: "Ingo Reimann" <ireimann@xxxxxxxxx>
An: "ceph-users" <ceph-users@xxxxxxx>
Gesendet: Mittwoch, 18. Dezember 2019 11:40:39
Betreff: radosgw - Etags suffixed with #x0e
Hi,
We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put).
I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws. At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types.
Any ideas?
best regards,
Ingo
--
Ingo Reimann
Teamleiter Technik
[ https://www.dunkel.de/ ]
Dunkel GmbH
Philipp-Reis-Straße 2
65795 Hattersheim
Fon: +49 6190 889-100
Fax: +49 6190 889-399
eMail: support@xxxxxxxxx
http://www.Dunkel.de/ Amtsgericht Frankfurt/Main
HRB: 37971
Geschäftsführer: Axel Dunkel
Ust-ID: DE 811622001
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
