Hi Cbodley , As you requested by IRC we tested directly with AWS Cli. Results: aws --endpoint=http://XXXXXXXXXX --profile=ceph s3api list-multipart-uploads --bucket Evol6 It reports near 170 uploads. We used the last one: { "Initiator": { "DisplayName": "xxxxx", "ID": "xxxxxx" }, "Initiated": "2019-12-03T01:23:06.007Z", "UploadId": "2~r0BMPPs8CewVZ6Qheu1s9WzaBn7bBvU", "StorageClass": "STANDARD", "Key": "MBS-da43656f-2b8c-464f-b341-03fdbdf446ae/CBB_SRV2K12/CBB_VM/192.168.0.197/SRV2K12/Hard disk 1$/20191203010516/431.cbrevision", "Owner": { "DisplayName": "xxxxx", "ID": "xxxx" } } aws --endpoint=http://xxxxxxxxx --profile=ceph s3api abort-multipart-upload --bucket Evol6 --key 'MBS-da43656f-2b8c-464f-b341-03fdbdf446ae/CBB_SRV2K12/CBB_VM/192.168.0.197/SRV2K12/Hard disk 1$/20191203010516/431.cbrevision' --upload-id 2~r0BMPPs8CewVZ6Qheu1s9WzaBn7bBvU Return: An error occurred (NoSuchUpload) when calling the AbortMultipartUpload operation: Unknown The same error is reported by S3CMD. Maybe is there something wrong parsing the "1$" inside the key Best Regards, Regards Manuel -----Mensaje original----- De: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> En nombre de EDH - Manuel Rios Enviado el: martes, 21 de enero de 2020 20:09 Para: Robin H. Johnson <robbat2@xxxxxxxxxx> CC: ceph-users@xxxxxxxxxxxxxx Asunto: Re: S3 Bucket usage up 150% diference between rgw-admin and external metering tools. Hi Robin, - What are the external tools? CloudBerry S3 Explorer and S3 Browser - How many objects do the external tools report as existing? Tool report 72142 keys (Aprox 6TB) vs CEPH num_objects 180981 (9TB) - Do the external tools include incomplete multipart uploads in their size data? I think no one external software include incomplete objects in the size, due S3 api list recursive don't include it. Checking for incomplete multiparts , I got a response 404 NoSuchKeys. - If bucket versioning is enabled, do the tools include all versions in the size data? Versioning is not enabled - Are there leftover multipart pieces without a multipart head? How can we check it? Specific bucket information: { "bucket": "XXXXXX", "tenant": "", "zonegroup": "4d8c7c5f-ca40-4ee3-b5bb-b2cad90bd007", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "default.rgw.buckets.data", "data_extra_pool": "default.rgw.buckets.non-ec", "index_pool": "default.rgw.buckets.index" }, "id": "48efb8c3-693c-4fe0-bbe4-fdc16f590a82.132873679.2", "marker": "48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.52", "index_type": "Normal", "owner": "XXXXXXX", "ver": "0#89789,1#60165,2#80652,3#76367", "master_ver": "0#0,1#0,2#0,3#0", "mtime": "2020-01-05 19:29:59.360574Z", "max_marker": "0#,1#,2#,3#", "usage": { "rgw.main": { "size": 9050249319344, "size_actual": 9050421526528, "size_utilized": 9050249319344, "size_kb": 8838134101, "size_kb_actual": 8838302272, "size_kb_utilized": 8838134101, "num_objects": 180981 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 3861, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 4, "num_objects": 143 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1024, "max_size_kb": 0, "max_objects": -1 } } -----Mensaje original----- De: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> En nombre de Robin H. Johnson Enviado el: martes, 21 de enero de 2020 18:58 CC: ceph-users@xxxxxxxxxxxxxx Asunto: Re: S3 Bucket usage up 150% diference between rgw-admin and external metering tools. On Mon, Jan 20, 2020 at 12:57:51PM +0000, EDH - Manuel Rios wrote: > Hi Cephs > > Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage / backups information. > > Today checking the data usage with a customer found that rgw-admin is reporting: ... > That's near 5TB used space in CEPH, and the external tools are reporting just 1.42TB. - What are the external tools? - How many objects do the external tools report as existing? - Do the external tools include incomplete multipart uploads in their size data? - If bucket versioning is enabled, do the tools include all versions in the size data? - Are there leftover multipart pieces without a multipart head? (this is a Ceph bug that I think is fixed in your release, but old pieces might still exist). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2@xxxxxxxxxx GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |