Hi,Our Ceph cluster is a 6 Node cluster each node having 8 disks. The cluster is used for object storage only (right now). We do use EC 3+2 on the buckets.data pool.
We had a problem with RadosGW segfaulting (12.2.5) till we upgraded to 12.2.8. We had almost 30.000 radosgw crashes leading to millions of unreferenced objects (failed multiuploads?). It filled our cluster so fast that we are now in danger to run out of space.
As you can see we are reweighting some OSDs right now. But the real question is how "used" is calculated in ceph df.
Global: %RAW USED = 76.49% while x-1.rgw.buckets.data Used = 90.32% Am I right this is because we should still be "able" to loose one OSD node?If thats true - reweight can help just a little to rebalance the capacity used on each node?
The only chance we have right now to survive until new HDDs arrive is to delete objects, right?
ceph -s
cluster:
id: a2222146-6561-307e-b032-xxxxxxxxxxxxx
health: HEALTH_WARN
3 nearfull osd(s)
13 pool(s) nearfull
1 large omap objects
766760/180478374 objects misplaced (0.425%)
services:
mon: 3 daemons, quorum ceph1-mon3,ceph1-mon2,ceph1-mon1
mgr: ceph1-mon2(active), standbys: ceph1-mon1, ceph1-mon3
osd: 36 osds: 36 up, 36 in; 24 remapped pgs
rgw: 3 daemons active
rgw-nfs: 2 daemons active
data:
pools: 13 pools, 1424 pgs
objects: 36.10M objects, 115TiB
usage: 200TiB used, 61.6TiB / 262TiB avail
pgs: 766760/180478374 objects misplaced (0.425%)
1400 active+clean
16 active+remapped+backfill_wait
8 active+remapped+backfilling
io:
client: 3.05MiB/s rd, 0B/s wr, 1.12kop/s rd, 37op/s wr
recovery: 306MiB/s, 91objects/s
ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
262TiB 61.6TiB 200TiB 76.49
POOLS:
NAME ID USED %USED MAX AVAIL
OBJECTS
iscsi-images 1 35B 0 6.87TiB
5
.rgw.root 2 3.57KiB 0 6.87TiB
18
x-1.rgw.buckets.data 6 115TiB 90.32 12.4TiB
36090523
x-1.rgw.control 7 0B 0 6.87TiB
8
x-1.rgw.meta 8 943KiB 0 6.87TiB
3265
x-1.rgw.log 9 0B 0 6.87TiB
407
x-1.rgw.buckets.index 12 0B 0 6.87TiB
3096
x-1.rgw.buckets.non-ec 13 0B 0 6.87TiB
1623
default.rgw.meta 14 373B 0 6.87TiB
3
default.rgw.control 15 0B 0 6.87TiB
8
default.rgw.log 16 0B 0 6.87TiB
0
scbench 17 0B 0 6.87TiB
0
rbdbench 18 1.00GiB 0.01 6.87TiB
260
Regards, Flo
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
