Hi,
I've never heard of automatic data deletion. Maybe just some snapshots
were removed? Or someone deleted data on purpose because of the
nearfull state of some OSDs? And there's no trash function for cephfs
(for rbd there is). Do you use cephfs snapshots?
Zitat von Prabu GJ <gjprabu@xxxxxxxxxxxx>:
Hi Team,
We are using Ceph Octopus version with a total disk size of 136 TB,
configured with two replicas. Currently, our usage is 57 TB, and the
available size is 5.3 TB. An incident occurred yesterday where
around 3 TB of data was deleted automatically. Upon analysis, we
couldn't find the reason for the deletion. All OSDs are functioning
properly and actively running.
We have 3 MDS , we try to restarted all MDS services. Is there any
solution to recover those data. Can anyone please help us find the
issue?
cluster:
id: 0d605d58-5caf-4f76-b6bd-e12402a22296
health: HEALTH_WARN
insufficient standby MDS daemons available
5 nearfull osd(s)
3 pool(s) nearfull
1 pool(s) have non-power-of-two pg_num
services:
mon: 4 daemons, quorum
download-mon3,download-mon4,download-mon1,download-mon2 (age 14h)
mgr: download-mon2(active, since 14h), standbys: download-mon1,
download-mon3
mds: integdownload:2
{0=download-mds3=up:active,1=download-mds1=up:active}
osd: 39 osds: 39 up (since 16h), 39 in (since 4d)
data:
pools: 3 pools, 1087 pgs
objects: 71.76M objects, 51 TiB
usage: 105 TiB used, 31 TiB / 136 TiB avail
pgs: 1087 active+clean
io:
client: 414 MiB/s rd, 219 MiB/s wr, 513 op/s rd, 1.22k op/s wr
================================================================
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA
STATE
0 download-osd1 2995G 581G 14 4785k 6 6626k
exists,up
1 download-osd2 2578G 998G 84 3644k 18 10.1M
exists,up
2 download-osd3 3093G 483G 17 5114k 5 4152k
exists,nearfull,up
3 download-osd4 2757G 819G 12 996k 2 4107k
exists,up
4 download-osd5 2889G 687G 28 3355k 20 8660k
exists,up
5 download-osd6 2448G 1128G 183 3312k 10 9435k
exists,up
6 download-osd7 2814G 762G 7 1667k 4 6354k
exists,up
7 download-osd8 2872G 703G 14 1672k 15 10.5M
exists,up
8 download-osd9 2577G 999G 10 6615k 3 6960k
exists,up
9 download-osd10 2651G 924G 16 4736k 3 7378k
exists,up
10 download-osd11 2889G 687G 15 4810k 6 8980k
exists,up
11 download-osd12 2912G 664G 11 2516k 2 4106k
exists,up
12 download-osd13 2785G 791G 74 4643k 11 3717k
exists,up
13 download-osd14 3150G 426G 214 6133k 4 7389k
exists,nearfull,up
14 download-osd15 2728G 848G 11 4959k 4 6603k
exists,up
15 download-osd16 2682G 894G 13 3170k 3 2503k
exists,up
16 download-osd17 2555G 1021G 53 2183k 7 5058k
exists,up
17 download-osd18 3013G 563G 18 3497k 3 4427k
exists,up
18 download-osd19 2924G 651G 24 3534k 12 10.4M
exists,up
19 download-osd20 3003G 573G 19 5149k 3 2531k
exists,up
20 download-osd21 2757G 819G 16 3707k 9 9816k
exists,up
21 download-osd22 2576G 999G 15 2526k 8 7739k
exists,up
22 download-osd23 2758G 818G 13 4412k 16 7125k
exists,up
23 download-osd24 2862G 714G 18 4424k 6 5787k
exists,up
24 download-osd25 2792G 783G 16 1972k 9 9749k
exists,up
25 download-osd26 2397G 1179G 14 4296k 9 12.0M
exists,up
26 download-osd27 2308G 1267G 8 3149k 22 6280k
exists,up
27 download-osd29 2732G 844G 12 3357k 3 7372k
exists,up
28 download-osd28 2814G 761G 11 476k 5 3316k
exists,up
29 download-osd30 3069G 507G 15 9043k 17 5628k
exists,nearfull,up
30 download-osd31 2660G 916G 15 841k 14 7798k
exists,up
31 download-osd32 2037G 1539G 10 1153k 15 3719k
exists,up
32 download-osd33 3116G 460G 20 7704k 12 9041k
exists,nearfull,up
33 download-osd34 2847G 728G 19 5788k 4 9014k
exists,up
34 download-osd35 3088G 488G 17 7178k 7 5730k
exists,nearfull,up
35 download-osd36 2414G 1161G 27 2017k 14 7612k
exists,up
36 download-osd37 2760G 815G 17 4292k 5 10.6M
exists,up
37 download-osd38 2679G 897G 12 2610k 5 10.0M
exists,up
38 download-osd39 3013G 563G 18 1804k 7 9235k
exists,up
Regards
Prabu GJ
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
