[RGW][Lifecycle][Versioned Buckets][Reef] Although LC deletes non-current versions, they still exist

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is similar to an old thread https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/FXVEWDU6NYCGEY5QB6IGGQXTUEZAQKNY/ but I don't see any responses there so opening this one.

PROBLEM DESCRIPTION

* Issue is seen on versioned buckets.
* Using extended logging (debug level 5), we can see that LC deletes expired (non-current) versions of objects but those versions are still found in the bucket: both seen in bucket index and accessible to the user.

AN EXAMPLE OBJECT/VERSION

* Bucket: bald
* Object: adc/certs/injr-f5lb01b-vadc04.json
* Versions: It currently has 6 versions

  $ aws s3api list-object-versions --bucket=bald --prefix=adc/certs/injr-f5lb01b-vadc04.json jq -r '.Versions[] | [.Key, .VersionId, .LastModified] | @tsv'
  adc/certs/injr-f5lb01b-vadc04.json      nBHrRDYZzuIrA0hORAIzh6QG8rzRF14  2024-06-28T21:13:00.014Z
  adc/certs/injr-f5lb01b-vadc04.json      YGgH7VmZDq4M-j8qIrKq.4Valvvuoh4   2024-06-26T20:58:09.835Z
  adc/certs/injr-f5lb01b-vadc04.json      qefseb1l.6WJyDNhH5buqX-qcZV2GAJ  2024-06-18T21:32:02.304Z
  adc/certs/injr-f5lb01b-vadc04.json      s4YG598JEQC9A5jaJuI5S4XkCh1NRpN 2024-06-10T21:37:16.074Z 
  adc/certs/injr-f5lb01b-vadc04.json      z96LISOi8jBYHCrnbqHgNAAsnpAqbXm 2024-06-07T01:15:21.802Z
  adc/certs/injr-f5lb01b-vadc04.json      Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp 2024-05-30T19:45:19.726Z

Looking at the oldest version with ID Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp ... LC actually deletes (or tries to delete?) it at several different occasions -- LC runs daily at midnight UTC. For example,
.
.
.
2024-07-04T00:00:02.711+0000 7fd30f989700  2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 1, 0
.
.
2024-07-08T00:00:04.989+0000 7f199a7ac700  2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 2, 3
.
.
2024-07-09T00:00:02.671+0000 7f5a23cea700  2 lifecycle: DELETED::bald[1eeb7b2c-aaab-4dff-be19-be27acab9e85.352350675.1034]):adc/certs/injr-f5lb01b-vadc04.json[Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp] (non-current expiration) wp_thrd: 0, 4

However, as seen in the above in aws-cli output, this version is still there. Below is the output when we retrieve this exact version:

$ aws s3api get-object --bucket=bald --key=adc/certs/injr-f5lb01b-vadc04.json --version-id=Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp /tmp/outfile
{
    "AcceptRanges": "bytes",
    "Expiration": "expiry-date=\"Sat, 01 Jun 2024 00:00:00 GMT\", rule-id=\"delete-prior-versions\"",
    "LastModified": "Thu, 30 May 2024 19:45:19 GMT",
    "ContentLength": 1299,
    "ETag": "\"d9c9ff538f4e2f1435746d16cd9e62c8\"",
    "VersionId": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp",
    "ContentType": "binary/octet-stream",
    "Metadata": {}
}

and in radosgw-admin bucket list:

    {
        "name": "adc/certs/injr-f5lb01b-vadc04.json",
        "instance": "Rfgi2NdGYWy.g7H6JevgqsDLXahSHJp",
        "ver": {
            "pool": 15,
            "epoch": 563937
        },
        "locator": "",
        "exists": true,
        "meta": {
            "category": 1,
            "size": 1299,
            "mtime": "2024-05-30T19:45:19.726701Z",
            "etag": "d9c9ff538f4e2f1435746d16cd9e62c8",
            "storage_class": "",
            ...
            "content_type": "",
            "accounted_size": 1299,
            "user_data": "",
            "appendable": false
        },
        "tag": "1eeb7b2c-aaab-4dff-be19-be27acab9e85.1228454711.3959044872362449539",
        "flags": 1,
        "pending_map": [],
        "versioned_epoch": 85
    },

SOME MORE NOTES ON THIS BUCKET & OBJECT

* The current object is not deleted: no delete-marker.
* No object locking is configured for this bucket.
* I don't see any trace of this bucket or object in gc list.
* The bucket has 101 shards and each shard has around ~30K objects so there's no noticeable skewness in distribution of objects across bucket index. However, I see below ERROR lines streaming when listing the bucket. Not sure it'd be relevant to the LC issue.
    ...
    2024-07-09T18:28:32.470+0000 7f8ef5db0740  0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=4, prev_marker=lb_summary.lock[s2BIhE0HnXnj.yONcP6.T-dkwU-aWhn], cur_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS]
    2024-07-09T18:28:32.530+0000 7f8ef5db0740  0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=2, prev_marker=lb_summary.lock[njFF6AkNKUoCVIsi-6pJVhDyaK8FycS], cur_marker=lb_summary.lock[GDUSDyTB4nGsjT0GfDYlnB5zrB8UnSV]
    2024-07-09T18:28:32.546+0000 7f8ef5db0740  0 ERROR: list_objects_ordered marker failed to make forward progress; attempt=3, prev_marker=lb_summary.lock[U4cpWSv2b5bI8rm1vsDw.kcXmXrDYuV], cur_marker=lb_summary.lock[2NCtXN7KbO0ypy4CSmJCk1gGZEnhWfL]
    ...

* We ran "bucket check --fix" on the bucket a few days ago but it didn't resolve the LC issue or "failed to make forward progress" error stream during bucket listing.
* Bucket stats for reference:
   $ radosgw-admin bucket stats --bucket=bald | jq '. | .bucket, [.num_shards, .usage]'
   "bald"
    [
      101,
      {
        "rgw.main": {
           "size": 13065121441,
            "size_actual": 16259469312,
            "size_utilized": 13065121441,
            "size_kb": 12758908,
             "size_kb_actual": 15878388,
             "size_kb_utilized": 12758908,
             "num_objects": 1233984
         }
      }
    ]

QUESTIONS

* Is this by  any chance a known issue? I searched the tracker but couldn't find a duplicate.
* Any ideas why the deletes initiated by LC might fail silently? I don't see any indication of gc queue getting full around the time.
* Any ideas on debugging this issue further? Would log-level 20 be helpful and/or any other log lines to look for?
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux