Hi!
I need some help understanding LC processing.
On latest versions of octopus installed (tested with 15.2.13 and 15.2.8) we
have at least one bucket which is not having the files removed when
expiring.
The size of the bucket reported with radosgw-admin compared with the one
obtained with s3cmd is different, logs below:
~3 TB in bucket stats and 11GB from s3cmd.
We tried to run manually several times the LC (lc process) but no success,
even with bucket check (including with --fix --check-objects) didn't help.
Configs changed are below, we have a few TB size buckets with millions of
objects, 6h default processing LC time was always not enough:
rgw_lc_debug_interval = 28800
rgw_lifecycle_work_time = 00:00-23:59
rgw_lc_max_worker = 5
rgw_lc_max_wp_worker = 9
rgw_enable_lc_threads = true
Status of LC is always complete:
{
"bucket":
":feeds-bucket-dev-dc418787:3ccb869f-b0f4-4fb9-a8d7-ecf5f5e18f33.37270170.140",
"started": "Mon, 26 Jul 2021 07:10:14 GMT",
"status": "COMPLETE"
},
It looks like versioning is not supported yet, and with aws-cli I did not
get any response for 's3api get-bucket-versioning'.
So, what to do?
$ s3cmd -c .s3cfg-feeds-bucket-dev-dc418787 du
s3://feeds-bucket-dev-dc418787
*12725854360 *192 objects s3://feeds-bucket-dev-dc418787/
(192 files, 11GB)
Output of 'radosgw-admin lc get' and 'bucket stats' is attached.
Thank you for any suggestion!
Also, we found that the LC is removing files earlier than set for another
bucket, 6h after the file was added instead of 31 days.
Paul
Sample of headers for an object, observe the expiry date (2 days, as in LC rule):
{
"AcceptRanges": "bytes",
"Expiration": "expiry-date=\"Mon, 12 Jul 2021 00:00:00 GMT\", rule-id=\"cleanup\"",
"LastModified": "Fri, 09 Jul 2021 10:52:23 GMT",
"ContentLength": 1274290121,
"ETag": "\"d5eac62748ba7fea103f1973e563966a-244\"",
"ContentType": "text/csv; charset=utf-8",
"Metadata": {}
}
# radosgw-admin bucket stats --bucket=feeds-bucket-dev-dc418787
2021-07-26T13:28:43.776+0300 7fddc7df0080 2 all 8 watchers are set, enabling cache
2021-07-26T13:28:43.776+0300 7fdd88fe9700 2 RGWDataChangesLog::ChangesRenewThread: start
{
"bucket": "feeds-bucket-dev-dc418787",
"num_shards": 30,
"tenant": "",
"zonegroup": "93c4027b-e930-473e-88e4-341ca31cfc2c",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "3ccb869f-b0f4-4fb9-a8d7-ecf5f5e18f33.46261325.1",
"marker": "3ccb869f-b0f4-4fb9-a8d7-ecf5f5e18f33.37270170.140",
"index_type": "Normal",
"owner": "user-dc418787",
"ver": "0#7328,1#28720,2#239,3#170,4#46497,5#6412,6#4894,7#9952,8#24841,9#22430,10#57475,11#41288,12#69788,13#44486,14#10091,15#50822,16#1364,17#50524,18#6216,19#156400,20#8052,21#660,22#42518,23#4178,24#12188,25#564,26#8123,27#548,28#42203,29#70443",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0",
"mtime": "2021-07-26T06:12:23.448315Z",
"creation_time": "2020-07-08T11:01:01.929011Z",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#",
"usage": {
"rgw.main": {
"size": 3302827555820,
"size_actual": 3302828068864,
"size_utilized": 3302827555820,
"size_kb": 3225417535,
"size_kb_actual": 3225418036,
"size_kb_utilized": 3225417535,
"num_objects": 627741
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 34101,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 34,
"num_objects": 1263
# radosgw-admin lc get --bucket=feeds-bucket-dev-dc418787
2021-07-26T13:38:37.177+0300 7fb5a3d44080 2 all 8 watchers are set, enabling cache
2021-07-26T13:38:37.181+0300 7fb565fdb700 2 RGWDataChangesLog::ChangesRenewThread: start
{
"prefix_map": {
"": {
"status": true,
"dm_expiration": false,
"expiration": 2,
"noncur_expiration": 0,
"mp_expiration": 1,
"transitions": {},
"noncur_transitions": {}
}
},
"rule_map": [
{
"id": "cleanup",
"rule": {
"id": "cleanup",
"prefix": "",
"status": "Enabled",
"expiration": {
"days": "2",
"date": ""
},
"noncur_expiration": {
"days": "",
"date": ""
},
"mp_expiration": {
"days": "1",
"date": ""
},
"filter": {
"prefix": "",
"obj_tags": {
"tagset": {}
}
},
"transitions": {},
"noncur_transitions": {},
"dm_expiration": false
}
}
]
}
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
