Igor,
Yes. Just a bit.
root@pmx101:/mnt/pve/iso# du -h | wc -l
10
root@pmx101:/mnt/pve/iso# du -h
0 ./snippets
0 ./tmp
257M ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2
5.5T ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14
5.5T ./xcp_nfs_sr
0 ./failover_test
11G ./template/iso
11G ./template
0 ./xcpiso
5.5T .
root@pmx101:/mnt/pve/iso# du --inodes
1 ./snippets
1 ./tmp
5 ./xcp_nfs_sr/2ba36cf5-291a-17d2-b510-db1a295ce0c2
53 ./xcp_nfs_sr/5aacaebb-4469-96f9-729e-fe45eef06a14
59 ./xcp_nfs_sr
1 ./failover_test
2 ./template/iso
3 ./template
1 ./xcpiso
67 .
root@pmx101:/mnt/pve/iso# rados lssnap -p cephfs.shared.data
0 snaps
What/where are all the other objects?!?
On 15/03/2024 3:36 am, Igor Fedotov wrote:
Thorn,
you might want to assess amount of files on the mounted fs by runnning
"du -h | wc". Does it differ drastically from amount of objects in the
pool = ~3.8 M?
And just in case - please run "rados lssnap -p cephfs.shared.data".
Thanks,
Igor
On 3/14/2024 1:42 AM, Thorne Lawler wrote:
Igor, Etienne, Bogdan,
The system is a four node cluster. Each node has 12 3.8TB SSDs, and
each SSD is an OSD.
I have not defined any separate DB / WAL devices - this cluster is
mostly at cephadm defaults.
Everything is currently configured to have x3 replicas.
The system also does various RBD workloads from other pools.
There are no subvolumes and no snapshots on the CephFS volume in
question.
The CephFS volume I am concerned about is called 'shared'. For the
purposes of this question I am omitting information about the other
pools.
[root@san1 ~]# rados df
POOL_NAME USED OBJECTS CLONES COPIES
MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD
WR_OPS WR USED COMPR UNDER COMPR
cephfs.shared.data 41 TiB 3834689 0
11504067 0 0 0 3219785418 175 TiB
9330001764 229 TiB 7.0 MiB 12 MiB
cephfs.shared.meta 757 MiB 85 0
255 0 0 0 5306018840 26 TiB
9170232158 24 TiB 0 B 0 B
total_objects 13169948
total_used 132 TiB
total_avail 33 TiB
total_space 166 TiB
[root@san1 ~]# ceph df detail
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 166 TiB 33 TiB 132 TiB 132 TiB 79.82
TOTAL 166 TiB 33 TiB 132 TiB 132 TiB 79.82
--- POOLS ---
POOL ID PGS STORED (DATA) (OMAP)
OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS
QUOTA BYTES DIRTY USED COMPR UNDER COMPR
cephfs.shared.meta 3 32 251 MiB 208 MiB 42 MiB
84 752 MiB 625 MiB 127 MiB 0 3.4 TiB
N/A N/A N/A 0 B 0 B
cephfs.shared.data 4 512 14 TiB 14 TiB 0 B
3.83M 41 TiB 41 TiB 0 B 79.90 3.4 TiB
N/A N/A N/A 7.0 MiB 12 MiB
[root@san1 ~]# ceph osd pool get cephfs.shared.data size
size: 3
...however running 'du' in the root directory of the 'shared' volume
says:
# du -sh .
5.5T .
So yeah - 14TB is replicated to 41TB, that's fine, but 14TB is a lot
more than 5.5TB, so... where is that space going?
On 14/03/2024 2:09 am, Igor Fedotov wrote:
Hi Thorn,
could you please share the output of "ceph df detail" command
representing the problem?
And please give an overview of your OSD layout - amount of OSDs,
shared or dedicated DB/WAL, main and DB volume sizes.
Thanks,
Igor
On 3/13/2024 5:58 AM, Thorne Lawler wrote:
Hi everyone!
My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB
usage for the data pool, but there are only 5.5TB of files in it.
There are fewer than 100 files on the filesystem in total, so where
is all that space going?
How can I analyze my cephfs to understand what is using that space,
and if possible, how can I reclaim that space?
Thank you.
--
Regards,
Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard
ITGOV40172
P +61 499 449 170
_DDNS
/_*Please note:* The information contained in this email message and
any attached files may be confidential information, and may also be
the subject of legal professional privilege. _If you are not the
intended recipient any use, disclosure or copying of this email is
unauthorised. _If you received this email in error, please notify
Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this
matter and delete all copies of this transmission together with any
attachments. /
--
Igor Fedotov
Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
--
Regards,
Thorne Lawler - Senior System Administrator
*DDNS* | ABN 76 088 607 265
First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172
P +61 499 449 170
_DDNS
/_*Please note:* The information contained in this email message and any
attached files may be confidential information, and may also be the
subject of legal professional privilege. _If you are not the intended
recipient any use, disclosure or copying of this email is unauthorised.
_If you received this email in error, please notify Discount Domain Name
Services Pty Ltd on 03 9815 6868 to report this matter and delete all
copies of this transmission together with any attachments. /
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
