We are using Ceph Octopus version with a total disk size of 136 TB, configured with two replicas. Currently, our usage is 57 TB, and the available size is 5.3 TB. An incident occurred yesterday where around 3 TB of data was deleted automatically. Upon analysis, we couldn't find the reason for the deletion. All OSDs are functioning properly and actively running.
We have 3 MDS , we try to restarted all MDS services. Is there any solution to recover those data. Can anyone please help us find the issue?
cluster:
id: 0d605d58-5caf-4f76-b6bd-e12402a22296
health: HEALTH_WARN
insufficient standby MDS daemons available
5 nearfull osd(s)
3 pool(s) nearfull
1 pool(s) have non-power-of-two pg_num
services:
mon: 4 daemons, quorum download-mon3,download-mon4,download-mon1,download-mon2 (age 14h)
mgr: download-mon2(active, since 14h), standbys: download-mon1, download-mon3
mds: integdownload:2 {0=download-mds3=up:active,1=download-mds1=up:active}
osd: 39 osds: 39 up (since 16h), 39 in (since 4d)
data:
pools: 3 pools, 1087 pgs
objects: 71.76M objects, 51 TiB
usage: 105 TiB used, 31 TiB / 136 TiB avail
pgs: 1087 active+clean
io:
client: 414 MiB/s rd, 219 MiB/s wr, 513 op/s rd, 1.22k op/s wr
================================================================
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 download-osd1 2995G 581G 14 4785k 6 6626k exists,up
1 download-osd2 2578G 998G 84 3644k 18 10.1M exists,up
2 download-osd3 3093G 483G 17 5114k 5 4152k exists,nearfull,up
3 download-osd4 2757G 819G 12 996k 2 4107k exists,up
4 download-osd5 2889G 687G 28 3355k 20 8660k exists,up
5 download-osd6 2448G 1128G 183 3312k 10 9435k exists,up
6 download-osd7 2814G 762G 7 1667k 4 6354k exists,up
7 download-osd8 2872G 703G 14 1672k 15 10.5M exists,up
8 download-osd9 2577G 999G 10 6615k 3 6960k exists,up
9 download-osd10 2651G 924G 16 4736k 3 7378k exists,up
10 download-osd11 2889G 687G 15 4810k 6 8980k exists,up
11 download-osd12 2912G 664G 11 2516k 2 4106k exists,up
12 download-osd13 2785G 791G 74 4643k 11 3717k exists,up
13 download-osd14 3150G 426G 214 6133k 4 7389k exists,nearfull,up
14 download-osd15 2728G 848G 11 4959k 4 6603k exists,up
15 download-osd16 2682G 894G 13 3170k 3 2503k exists,up
16 download-osd17 2555G 1021G 53 2183k 7 5058k exists,up
17 download-osd18 3013G 563G 18 3497k 3 4427k exists,up
18 download-osd19 2924G 651G 24 3534k 12 10.4M exists,up
19 download-osd20 3003G 573G 19 5149k 3 2531k exists,up
20 download-osd21 2757G 819G 16 3707k 9 9816k exists,up
21 download-osd22 2576G 999G 15 2526k 8 7739k exists,up
22 download-osd23 2758G 818G 13 4412k 16 7125k exists,up
23 download-osd24 2862G 714G 18 4424k 6 5787k exists,up
24 download-osd25 2792G 783G 16 1972k 9 9749k exists,up
25 download-osd26 2397G 1179G 14 4296k 9 12.0M exists,up
26 download-osd27 2308G 1267G 8 3149k 22 6280k exists,up
27 download-osd29 2732G 844G 12 3357k 3 7372k exists,up
28 download-osd28 2814G 761G 11 476k 5 3316k exists,up
29 download-osd30 3069G 507G 15 9043k 17 5628k exists,nearfull,up
30 download-osd31 2660G 916G 15 841k 14 7798k exists,up
31 download-osd32 2037G 1539G 10 1153k 15 3719k exists,up
32 download-osd33 3116G 460G 20 7704k 12 9041k exists,nearfull,up
33 download-osd34 2847G 728G 19 5788k 4 9014k exists,up
34 download-osd35 3088G 488G 17 7178k 7 5730k exists,nearfull,up
35 download-osd36 2414G 1161G 27 2017k 14 7612k exists,up
36 download-osd37 2760G 815G 17 4292k 5 10.6M exists,up
37 download-osd38 2679G 897G 12 2610k 5 10.0M exists,up
38 download-osd39 3013G 563G 18 1804k 7 9235k exists,up
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
