Hi Thorn,
so the problem is apparently bound to huge file sizes. I presume they're
split into multiple chunks at ceph side hence producing millions of objects.
And possibly something is wrong with this mapping.
If this pool has no write load at the moment you might want to run the
following experiment:
1) put one more huge file to the filesystem, e.g. of 200GB bytes. And
note pool stats (through "ceph df detail --format json-pretty) before
and after this operation.
2) then remove the file, wait until object count is stabilized (i.e.
removal is completed) and learn the final stats.
Are there any leaks? What was stored space (object count) increase in
the middle of the above procedure?
As it looks like compession is (was?) enabled on the pool-in-question
it's worth using non-compressible data for the experiment, e.g. generate
them through /dev/urandom.
Thanks,
Igor
On 3/15/2024 2:05 AM, Thorne Lawler wrote:
Igor,
Yes. Just a bit.
root@pmx101:/mnt/pve/iso# du -h | wc -l
10
root@pmx101:/mnt/pve/iso# du -h
0 ./snippets
0 ./tmp
257M ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2
5.5T ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14
5.5T ./xcp_nfs_sr
0 ./failover_test
11G ./template/iso
11G ./template
0 ./xcpiso
5.5T .
root@pmx101:/mnt/pve/iso# du --inodes
1 ./snippets
1 ./tmp
5 ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2
53 ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14
59 ./xcp_nfs_sr
1 ./failover_test
2 ./template/iso
3 ./template
1 ./xcpiso
67 .
root@pmx101:/mnt/pve/iso# rados lssnap -p cephfs.shared.data
0 snaps
What/where are all the other objects?!?
On 15/03/2024 3:36 am, Igor Fedotov wrote:
Thorn,
you might want to assess amount of files on the mounted fs by
runnning "du -h | wc". Does it differ drastically from amount of
objects in the pool = ~3.8 M?
And just in case - please run "rados lssnap -p cephfs.shared.data".
Thanks,
Igor
On 3/14/2024 1:42 AM, Thorne Lawler wrote:
Igor, Etienne, Bogdan,
The system is a four node cluster. Each node has 12 3.8TB SSDs, and
each SSD is an OSD.
I have not defined any separate DB / WAL devices - this cluster is
mostly at cephadm defaults.
Everything is currently configured to have x3 replicas.
The system also does various RBD workloads from other pools.
There are no subvolumes and no snapshots on the CephFS volume in
question.
The CephFS volume I am concerned about is called 'shared'. For the
purposes of this question I am omitting information about the other
pools.
[root@san1 ~]# rados df
POOL_NAME USED OBJECTS CLONES COPIES
MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD
WR_OPS WR USED COMPR UNDER COMPR
cephfs.shared.data 41 TiB 3834689 0
11504067 0 0 0 3219785418 175
TiB 9330001764 229 TiB 7.0 MiB 12 MiB
cephfs.shared.meta 757 MiB 85 0
255 0 0 0 5306018840 26 TiB
9170232158 24 TiB 0 B 0 B
total_objects 13169948
total_used 132 TiB
total_avail 33 TiB
total_space 166 TiB
[root@san1 ~]# ceph df detail
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 166 TiB 33 TiB 132 TiB 132 TiB 79.82
TOTAL 166 TiB 33 TiB 132 TiB 132 TiB 79.82
--- POOLS ---
POOL ID PGS STORED (DATA) (OMAP)
OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS
QUOTA BYTES DIRTY USED COMPR UNDER COMPR
cephfs.shared.meta 3 32 251 MiB 208 MiB 42 MiB
84 752 MiB 625 MiB 127 MiB 0 3.4 TiB
N/A N/A N/A 0 B 0 B
cephfs.shared.data 4 512 14 TiB 14 TiB 0 B 3.83M
41 TiB 41 TiB 0 B 79.90 3.4 TiB N/A
N/A N/A 7.0 MiB 12 MiB
[root@san1 ~]# ceph osd pool get cephfs.shared.data size
size: 3
...however running 'du' in the root directory of the 'shared' volume
says:
# du -sh .
5.5T .
So yeah - 14TB is replicated to 41TB, that's fine, but 14TB is a lot
more than 5.5TB, so... where is that space going?
On 14/03/2024 2:09 am, Igor Fedotov wrote:
Hi Thorn,
could you please share the output of "ceph df detail" command
representing the problem?
And please give an overview of your OSD layout - amount of OSDs,
shared or dedicated DB/WAL, main and DB volume sizes.
Thanks,
Igor
On 3/13/2024 5:58 AM, Thorne Lawler wrote:
Hi everyone!
My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB
usage for the data pool, but there are only 5.5TB of files in it.
There are fewer than 100 files on the filesystem in total, so
where is all that space going?
How can I analyze my cephfs to understand what is using that
space, and if possible, how can I reclaim that space?
Thank you.
--
Regards,
Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard
ITGOV40172
P +61 499 449 170
_DDNS
/_*Please note:* The information contained in this email message and
any attached files may be confidential information, and may also be
the subject of legal professional privilege. _If you are not the
intended recipient any use, disclosure or copying of this email is
unauthorised. _If you received this email in error, please notify
Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this
matter and delete all copies of this transmission together with any
attachments. /
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
--
Regards,
Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170
_DDNS
/_*Please note:* The information contained in this email message and
any attached files may be confidential information, and may also be
the subject of legal professional privilege. _If you are not the
intended recipient any use, disclosure or copying of this email is
unauthorised. _If you received this email in error, please notify
Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this
matter and delete all copies of this transmission together with any
attachments. /
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx