Re: rgw bug adding null characters in multipart object names and in Etags

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi All



ok first problem (with mpu's) solved.

'--omap-key-file file' is doing the trick:


cat -A key.bin
_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC^@.25
rados -p zh-1.rgw.buckets.index rmomapkey .dir.cb1594b3-a782-49d0-a19f-68cd48870a63.81880353.1.0 --omap-key-file key.bin

cf. also:
https://tracker.ceph.com/issues/49018

https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/MQICUW7GACUYOSURJQKWRLLNNORA6CGD/



Now I am still asking for help to get rid of the nucll characters in rgw object etags...

Do you know if following commands may help and are safe in multi-site?
  • radosgw-admin bucket check --bucket $bucket --fix --check-objects
  • radosgw-admin bucket rewrite --bucket $bucket --min-rewrite-size 0



Cheers
Francois


--


EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@xxxxxxxxxxxx
web: http://www.everyware.ch


From: Scheurer François
Sent: Thursday, May 13, 2021 2:36 PM
To: ceph-users@xxxxxxx
Subject: Re: rgw bug adding null characters in multipart object names and in Etags
 

Hi All



listomapkeys is actually dealing correctly with the null chars and output them.

rmomapkey is not, but rados has a new option that should help:

--omap-key-file file

Read the omap key from a file.


Testing that...



Cheers

Francois



--


EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@xxxxxxxxxxxx
web: http://www.everyware.ch
From: Scheurer François <francois.scheurer@xxxxxxxxxxxx>
Sent: Thursday, May 13, 2021 12:09:12 PM
To: ceph-users@xxxxxxx
Subject: Re: rgw bug adding null characters in multipart object names and in Etags
 

Hi All



This trackers seems to be the one regarding the null characters in multipart object names:

https://tracker.ceph.com/issues/48874

So it is apparently fixed in 14.2.17, but how to remove invalid omap keys that were added previously?

The 'rados' command is not converting \0 or \\0 in null chars .

Is there another way to edit the omap keys?
Will some one enhance the 'rados' command for this?


Cheers
Francois


--


EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@xxxxxxxxxxxx
web: http://www.everyware.ch
From: Scheurer François <francois.scheurer@xxxxxxxxxxxx>
Sent: Saturday, May 8, 2021 12:09:14 PM
To: ceph-users@xxxxxxx
Subject: rgw bug adding null characters in multipart object names and in Etags
 

Dear All



We are trying to remove old multipart uploads but get in trouble with some of them having null characters:


rados -p zh-1.rgw.buckets.index rmomapkey .dir.cb1594b3-a782-49d0-a19f-68cd48870a63.81880353.1.0 '_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC.25'


rados -p zh-1.rgw.buckets.index rmomapkey .dir.cb1594b3-a782-49d0-a19f-68cd48870a63.81880353.1.0 $(echo -ne '_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC\0.25')
-bash: warning: command substitution: ignored null byte in input

rados -p zh-1.rgw.buckets.index listomapkeys .dir.cb1594b3-a782-49d0-a19f-68cd48870a63.81880353.1.0 | grep -a '_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC' | cat -A
_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC^@.25$ # <= not deleted !

It is not working, as the Null Char is stripped off.
Any Idea how to proceed?

This bucket was created on luminous. But this specific object was created after our upgrade to nautilus.
Apparently some bugs have added NullChars at the end of MPU object names, between uploadid and suffix.
Output from 'radosgw-admin bi list' (see the \u0000 NullChars):
  {
    "type": "plain",
    "idx": "_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC\u0000.25",
    "entry": {
      "name": "_multipart_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_DiskImage/Disk_4f8130ff-fef5-4b0f-b25e-c6b8b3dba9bf/Volume_NTFS_5b4f5274-9107-4386-93d9-e7f31193805a$/20201218230243/0.cbrevision.525Sr39KY5yVbD_w9ipOXSXsQ95YUnC\u0000.25",
      "instance": "",
      "ver": {
        "pool": 6,
        "epoch": 852938
      },
      "locator": "",
      "exists": "true",
      "meta": {
        "category": 1,
        "size": 157286400,
        "mtime": "2020-12-25 23:39:20.019898Z",
        "etag": "a126c2f0d439c44176a5d07bd5841575",
        "storage_class": "",
        "owner": "40eb21a9092c4948bcf94386f6042f94",
        "owner_display_name": "amsler1",
        "content_type": "",
        "accounted_size": 157286400,
        "user_data": "",
        "appendable": "false"
      },
      "tag": "_vMx_4vu-E5nWf7kCHJIQCFPGEHRiUAG",
      "flags": 0,
      "pending_map": [],
      "versioned_epoch": 0
    }
  },


On the same bucket, we also see NullChars at the end of some Etags when we using 'radosgw-admin bucket list --bucket' but not with 'radosgw-admin object stat':

object='MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml'

radosgw-admin object stat --bucket="$bucket" --object="$object" | jq -c '{name, size, etag, tag, obj_size: .manifest.obj_size, marker:.manifest.tail_placement.bucket.marker, bucket_id:.manifest.tail_placement.bucket.bucket_id}' | cat -A
{"name":"MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml","size":39372,"etag":"0e73d594032900acb74d3f06b230aeb9","tag":"_xhNKxuWrfxDO5XfYs8Llq8vLTUYqtmm","obj_size":39372,"marker":"cb1594b3-a782-49d0-a19f-68cd48870a63.19334234.139","bucket_id":"cb1594b3-a782-49d0-a19f-68cd48870a63.20382694.169"}$ # <= no NullChar
radosgw-admin bucket list --bucket "${bucket}" --allow-unordered --max-entries 20000000 | jq -c 'sort_by(.bucket) | .[] | {name, accounted_size: .meta.accounted_size, etag: .meta.etag}' | fgrep -a 0e73d594032900acb74d3f06b230aeb9 | cat -A
{"name":"MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml","accounted_size":39372,"etag":"0e73d594032900acb74d3f06b230aeb9\u0000"}$ # <= no NullChar

rados -p zh-1.rgw.buckets.data stat 'cb1594b3-a782-49d0-a19f-68cd48870a63.19334234.139_'"$object"
zh-1.rgw.buckets.data/cb1594b3-a782-49d0-a19f-68cd48870a63.19334234.139_MBS-35a9b79c-f27d-44f2-804f-472ef0520816/CBB_BSSRV01/CBB_HV/BSSRV01.Aerztehaus-allschwil.ch/BSSRV05/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml:/20191103183115/D0F970B6-DB86-48AF-AA68-946D4642E2A6.xml mtime 2020-04-21 14:21:27.000000, size 39372

This bucket was causing multi-site rgw sync to crash every minute when using rgw_sync_obj_etag_verify = true.
These Etag NullChars may be the cause of this bug:
It may also be related to:


So we would be glad to know how to remove these NullChars from the Etags and how to remove the MPU's with NullChars in the object names...
These both issues seem to be the cause of many weird behaviors:
  1. rgw sync crashes (with rgw_sync_obj_etag_verify = true)
  2. radosgw-admin bucket sync status --bucket "$bucket" --source-zone ch-zh1-az2 => reports "bucket is caught up with source" but when most of the objects are missing
  3. radosgw-admin bucket list --bucket "$bucket" --allow-unordered --max-entries 99000000 => returns an imcomplete list
  4. radosgw-admin bucket stats --bucket "$bucket" => returns wrong number of objects and utilized size
The only reliable outputs is from bi list:
  • radosgw-admin bi list --bucket=$bucket | jq -cr 'map(select(.type == "plain" or .type == "instance") | .entry'

Do you know if following commands may help and are safe in multi-site?

  • radosgw-admin bucket check --bucket $bucket --fix --check-objects
  • radosgw-admin bucket rewrite --bucket $bucket --min-rewrite-size 0


Or maybe only a dedicated tool need to be developped to deal with these NullChars?

Many thanks in advance.



Cheers

Francois Scheurer



--


EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich

tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@xxxxxxxxxxxx
web: http://www.everyware.ch

Attachment: smime.p7s
Description: S/MIME cryptographic signature

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux