On Thu, Jan 10, 2019 at 4:07 PM Scottix <scottix@xxxxxxxxx> wrote:
I just had this question as well.
I am interested in what you mean by fullest, is it percentage wise or raw space. If I have an uneven distribution and adjusted it, would it make more space available potentially.
Yes - I'd recommend using pg-upmap if all your clients are Luminous+. I "reclaimed" about 5TB of usable space recently by balancing my PGs.
@Yoanne, you've got a fair bit of variance so you would likely benefit from pg-upmap (or other rebalancing).
Thanks
Scott_______________________________________________On Thu, Jan 10, 2019 at 12:05 AM Wido den Hollander <wido@xxxxxxxx> wrote:
On 1/9/19 2:33 PM, Yoann Moulin wrote:
> Hello,
>
> I have a CEPH cluster in luminous 12.2.10 dedicated to cephfs.
>
> The raw size is 65.5 TB, with a replica 3, I should have ~21.8 TB usable.
>
> But the size of the cephfs view by df is *only* 19 TB, is that normal ?
>
Yes. Ceph will calculate this based on the fullest OSD. As data
distribution is never 100% perfect you will get such numbers.
To go from raw to usable I use this calculation:
(RAW / 3) * 0.85
So yes, I take a 20%, sometimes even 30% buffer.
Wido
> Best regards,
>
> here some hopefully useful information :
>
>> apollo@icadmin004:~$ ceph -s
>> cluster:
>> id: fc76846a-d0f0-4866-ae6d-d442fc885469
>> health: HEALTH_OK
>>
>> services:
>> mon: 3 daemons, quorum icadmin006,icadmin007,icadmin008
>> mgr: icadmin006(active), standbys: icadmin007, icadmin008
>> mds: cephfs-3/3/3 up {0=icadmin008=up:active,1=icadmin007=up:active,2=icadmin006=up:active}
>> osd: 40 osds: 40 up, 40 in
>>
>> data:
>> pools: 2 pools, 2560 pgs
>> objects: 26.12M objects, 15.6TiB
>> usage: 49.7TiB used, 15.8TiB / 65.5TiB avail
>> pgs: 2560 active+clean
>>
>> io:
>> client: 510B/s rd, 24.1MiB/s wr, 0op/s rd, 35op/s wr
>
>> apollo@icadmin004:~$ ceph df
>> GLOBAL:
>> SIZE AVAIL RAW USED %RAW USED
>> 65.5TiB 15.8TiB 49.7TiB 75.94
>> POOLS:
>> NAME ID USED %USED MAX AVAIL OBJECTS
>> cephfs_data 1 15.6TiB 85.62 2.63TiB 25874848
>> cephfs_metadata 2 571MiB 0.02 2.63TiB 245778
>
>> apollo@icadmin004:~$ rados df
>> POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
>> cephfs_data 15.6TiB 25874848 0 77624544 0 0 0 324156851 25.9TiB 20114360 9.64TiB
>> cephfs_metadata 571MiB 245778 0 737334 0 0 0 1802713236 87.7TiB 75729412 16.0TiB
>>
>> total_objects 26120626
>> total_used 49.7TiB
>> total_avail 15.8TiB
>> total_space 65.5TiB
>
>> apollo@icadmin004:~$ ceph osd pool ls detail
>> pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 6197 lfor 0/3885 flags hashpspool stripe_width 0 application cephfs
>> pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 last_change 6197 lfor 0/703 flags hashpspool stripe_width 0 application cephfs
>
>> apollo@icadmin004:~$ df -h /apollo/
>> Filesystem Size Used Avail Use% Mounted on
>> 10.90.36.16,10.90.36.17,10.90.36.18:/ 19T 16T 2.7T 86% /apollo
>
>> apollo@icadmin004:~$ ceph fs get cephfs
>> Filesystem 'cephfs' (1)
>> fs_name cephfs
>> epoch 49277
>> flags c
>> created 2018-01-23 14:06:43.460773
>> modified 2019-01-09 14:17:08.520888
>> tableserver 0
>> root 0
>> session_timeout 60
>> session_autoclose 300
>> max_file_size 1099511627776
>> last_failure 0
>> last_failure_osd_epoch 6216
>> compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
>> max_mds 3
>> in 0,1,2
>> up {0=424203,1=424158,2=424146}
>> failed
>> damaged
>> stopped
>> data_pools [1]
>> metadata_pool 2
>> inline_data disabled
>> balancer
>> standby_count_wanted 0
>> 424203: 10.90.36.18:6800/3885954695 'icadmin008' mds.0.49202 up:active seq 6 export_targets=1,2
>> 424158: 10.90.36.17:6800/152758094 'icadmin007' mds.1.49198 up:active seq 16 export_targets=0,2
>> 424146: 10.90.36.16:6801/1771587593 'icadmin006' mds.2.49195 up:active seq 19 export_targets=0
>
>> apollo@icadmin004:~$ ceph osd tree
>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
>> -1 65.49561 root default
>> -7 3.27478 host iccluster150
>> 160 hdd 1.63739 osd.160 up 1.00000 1.00000
>> 165 hdd 1.63739 osd.165 up 1.00000 1.00000
>> -11 3.27478 host iccluster151
>> 163 hdd 1.63739 osd.163 up 1.00000 1.00000
>> 168 hdd 1.63739 osd.168 up 1.00000 1.00000
>> -5 3.27478 host iccluster152
>> 164 hdd 1.63739 osd.164 up 1.00000 1.00000
>> 169 hdd 1.63739 osd.169 up 1.00000 1.00000
>> -9 3.27478 host iccluster153
>> 162 hdd 1.63739 osd.162 up 1.00000 1.00000
>> 167 hdd 1.63739 osd.167 up 1.00000 1.00000
>> -3 3.27478 host iccluster154
>> 161 hdd 1.63739 osd.161 up 1.00000 1.00000
>> 166 hdd 1.63739 osd.166 up 1.00000 1.00000
>> -17 3.27478 host iccluster155
>> 170 hdd 1.63739 osd.170 up 1.00000 1.00000
>> 176 hdd 1.63739 osd.176 up 1.00000 1.00000
>> -21 3.27478 host iccluster156
>> 171 hdd 1.63739 osd.171 up 1.00000 1.00000
>> 177 hdd 1.63739 osd.177 up 1.00000 1.00000
>> -13 3.27478 host iccluster157
>> 172 hdd 1.63739 osd.172 up 1.00000 1.00000
>> 178 hdd 1.63739 osd.178 up 1.00000 1.00000
>> -15 3.27478 host iccluster158
>> 173 hdd 1.63739 osd.173 up 1.00000 1.00000
>> 179 hdd 1.63739 osd.179 up 0.90002 1.00000
>> -19 3.27478 host iccluster159
>> 174 hdd 1.63739 osd.174 up 0.95001 1.00000
>> 175 hdd 1.63739 osd.175 up 1.00000 1.00000
>> -23 3.27478 host iccluster160
>> 180 hdd 1.63739 osd.180 up 1.00000 1.00000
>> 185 hdd 1.63739 osd.185 up 1.00000 1.00000
>> -25 3.27478 host iccluster161
>> 181 hdd 1.63739 osd.181 up 1.00000 1.00000
>> 186 hdd 1.63739 osd.186 up 1.00000 1.00000
>> -27 3.27478 host iccluster162
>> 182 hdd 1.63739 osd.182 up 1.00000 1.00000
>> 187 hdd 1.63739 osd.187 up 1.00000 1.00000
>> -29 3.27478 host iccluster163
>> 183 hdd 1.63739 osd.183 up 1.00000 1.00000
>> 189 hdd 1.63739 osd.189 up 1.00000 1.00000
>> -31 3.27478 host iccluster164
>> 184 hdd 1.63739 osd.184 up 1.00000 1.00000
>> 188 hdd 1.63739 osd.188 up 1.00000 1.00000
>> -33 3.27478 host iccluster165
>> 190 hdd 1.63739 osd.190 up 1.00000 1.00000
>> 195 hdd 1.63739 osd.195 up 1.00000 1.00000
>> -35 3.27478 host iccluster166
>> 191 hdd 1.63739 osd.191 up 1.00000 1.00000
>> 197 hdd 1.63739 osd.197 up 1.00000 1.00000
>> -39 3.27478 host iccluster167
>> 192 hdd 1.63739 osd.192 up 1.00000 1.00000
>> 196 hdd 1.63739 osd.196 up 1.00000 1.00000
>> -37 3.27478 host iccluster168
>> 193 hdd 1.63739 osd.193 up 1.00000 1.00000
>> 198 hdd 1.63739 osd.198 up 1.00000 1.00000
>> -41 3.27478 host iccluster169
>> 194 hdd 1.63739 osd.194 up 1.00000 1.00000
>> 199 hdd 1.63739 osd.199 up 1.00000 1.00000
>
>> apollo@icadmin004:~$ ceph osd df
>> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
>> 160 hdd 1.63739 1.00000 1.64TiB 1.25TiB 393GiB 76.59 1.01 193
>> 165 hdd 1.63739 1.00000 1.64TiB 1.22TiB 431GiB 74.27 0.98 191
>> 163 hdd 1.63739 1.00000 1.64TiB 1.21TiB 440GiB 73.77 0.97 201
>> 168 hdd 1.63739 1.00000 1.64TiB 1.24TiB 409GiB 75.63 1.00 193
>> 164 hdd 1.63739 1.00000 1.64TiB 1.29TiB 357GiB 78.72 1.04 193
>> 169 hdd 1.63739 1.00000 1.64TiB 1.18TiB 471GiB 71.94 0.95 187
>> 162 hdd 1.63739 1.00000 1.64TiB 1.22TiB 432GiB 74.21 0.98 180
>> 167 hdd 1.63739 1.00000 1.64TiB 1.14TiB 514GiB 69.34 0.91 186
>> 161 hdd 1.63739 1.00000 1.64TiB 1.19TiB 460GiB 72.57 0.96 184
>> 166 hdd 1.63739 1.00000 1.64TiB 1.21TiB 439GiB 73.82 0.97 187
>> 170 hdd 1.63739 1.00000 1.64TiB 1.29TiB 354GiB 78.92 1.04 192
>> 176 hdd 1.63739 1.00000 1.64TiB 1.24TiB 408GiB 75.69 1.00 190
>> 171 hdd 1.63739 1.00000 1.64TiB 1.15TiB 496GiB 70.43 0.93 182
>> 177 hdd 1.63739 1.00000 1.64TiB 1.32TiB 325GiB 80.62 1.06 205
>> 172 hdd 1.63739 1.00000 1.64TiB 1.20TiB 451GiB 73.13 0.96 186
>> 178 hdd 1.63739 1.00000 1.64TiB 1.25TiB 400GiB 76.14 1.00 188
>> 173 hdd 1.63739 1.00000 1.64TiB 1.36TiB 285GiB 82.98 1.09 201
>> 179 hdd 1.63739 0.90002 1.64TiB 1.32TiB 327GiB 80.51 1.06 204
>> 174 hdd 1.63739 0.95001 1.64TiB 1.31TiB 332GiB 80.19 1.06 197
>> 175 hdd 1.63739 1.00000 1.64TiB 1.36TiB 286GiB 82.96 1.09 198
>> 180 hdd 1.63739 1.00000 1.64TiB 1.21TiB 433GiB 74.16 0.98 177
>> 185 hdd 1.63739 1.00000 1.64TiB 1.26TiB 391GiB 76.70 1.01 198
>> 181 hdd 1.63739 1.00000 1.64TiB 1.27TiB 380GiB 77.33 1.02 186
>> 186 hdd 1.63739 1.00000 1.64TiB 1.20TiB 451GiB 73.10 0.96 190
>> 182 hdd 1.63739 1.00000 1.64TiB 1.31TiB 332GiB 80.20 1.06 204
>> 187 hdd 1.63739 1.00000 1.64TiB 1.22TiB 424GiB 74.72 0.98 189
>> 183 hdd 1.63739 1.00000 1.64TiB 1.33TiB 318GiB 81.05 1.07 206
>> 189 hdd 1.63739 1.00000 1.64TiB 1.08TiB 576GiB 65.66 0.86 169
>> 184 hdd 1.63739 1.00000 1.64TiB 1.21TiB 441GiB 73.70 0.97 183
>> 188 hdd 1.63739 1.00000 1.64TiB 1.17TiB 474GiB 71.70 0.94 182
>> 190 hdd 1.63739 1.00000 1.64TiB 1.27TiB 373GiB 77.75 1.02 195
>> 195 hdd 1.63739 1.00000 1.64TiB 1.32TiB 327GiB 80.47 1.06 198
>> 191 hdd 1.63739 1.00000 1.64TiB 1.16TiB 484GiB 71.15 0.94 183
>> 197 hdd 1.63739 1.00000 1.64TiB 1.28TiB 370GiB 77.94 1.03 197
>> 192 hdd 1.63739 1.00000 1.64TiB 1.26TiB 382GiB 77.24 1.02 200
>> 196 hdd 1.63739 1.00000 1.64TiB 1.24TiB 402GiB 76.02 1.00 201
>> 193 hdd 1.63739 1.00000 1.64TiB 1.24TiB 409GiB 75.59 1.00 186
>> 198 hdd 1.63739 1.00000 1.64TiB 1.15TiB 501GiB 70.13 0.92 175
>> 194 hdd 1.63739 1.00000 1.64TiB 1.29TiB 353GiB 78.98 1.04 202
>> 199 hdd 1.63739 1.00000 1.64TiB 1.34TiB 309GiB 81.58 1.07 221
>> TOTAL 65.5TiB 49.7TiB 15.8TiB 75.94
>> MIN/MAX VAR: 0.86/1.09 STDDEV: 3.92
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
