When you use 3+2 EC that means you have 3 data chunks and 2 erasure chunks for your data. So you can handle two failures, but not three. The min_size setting is preventing you from going below 3 because that's the number of data chunks you specified for the pool. I'm sorry to say this, but since the data was wiped off the other 3 nodes there isn't anything that can be done to recover it.
Bryan
From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of Félix Barbeira <fbarbeira@xxxxxxxxx>
Date: Thursday, January 17, 2019 at 1:27 PM
To: Ceph Users <ceph-users@xxxxxxxxxxxxxx>
Subject: How to reduce min_size of an EC pool?
I want to bring back my cluster to HEALTHY state because right now I have not access to the data.
I have an 3+2 EC pool on a 5 node cluster. 3 nodes were lost, all data wiped. They were reinstalled and added to cluster again.
The "ceph health detail" command says to reduce min_size number to a value lower than 3, but:
root@ceph-monitor02:~# ceph osd pool set default.rgw.buckets.data min_size 2
Error EINVAL: pool min_size must be between 3 and 5
root@ceph-monitor02:~#
This is the situation:
root@ceph-monitor01:~# ceph -s
cluster:
id: ce78b02d-03df-4f9e-a35a-31b5f05c4c63
health: HEALTH_WARN
Reduced data availability: 515 pgs inactive, 512 pgs incomplete
services:
mon: 3 daemons, quorum ceph-monitor01,ceph-monitor03,ceph-monitor02
mgr: ceph-monitor02(active), standbys: ceph-monitor01, ceph-monitor03
osd: 57 osds: 57 up, 57 in
data:
pools: 8 pools, 568 pgs
objects: 4.48 M objects, 10 TiB
usage: 24 TiB used, 395 TiB / 419 TiB avail
pgs: 0.528% pgs unknown
90.141% pgs not active
512 incomplete
53 active+clean
3 unknown
root@ceph-monitor01:~#
And this is the output of health detail:
root@ceph-monitor01:~# ceph health detail
HEALTH_WARN Reduced data availability: 515 pgs inactive, 512 pgs incomplete
PG_AVAILABILITY Reduced data availability: 515 pgs inactive, 512 pgs incomplete
pg 10.1cd is stuck inactive since forever, current state incomplete, last acting [9,48,41,58,17] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1ce is incomplete, acting [3,13,14,42,21] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1cf is incomplete, acting [36,27,3,39,51] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d0 is incomplete, acting [29,9,38,4,56] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d1 is incomplete, acting [2,34,17,7,30] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d2 is incomplete, acting [41,45,53,13,32] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d3 is incomplete, acting [7,28,15,20,3] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d4 is incomplete, acting [11,40,25,23,0] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d5 is incomplete, acting [32,51,20,57,28] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d6 is incomplete, acting [2,53,8,16,15] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d7 is incomplete, acting [1,2,33,43,42] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d8 is incomplete, acting [27,49,9,48,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1d9 is incomplete, acting [37,8,7,11,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1da is incomplete, acting [27,14,33,15,53] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1db is incomplete, acting [58,53,6,26,4] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1dc is incomplete, acting [21,12,47,35,19] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1dd is incomplete, acting [51,4,52,24,7] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1de is incomplete, acting [38,29,21,41,44] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1df is incomplete, acting [25,4,30,61,11] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e0 is incomplete, acting [27,57,21,6,13] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e1 is incomplete, acting [8,7,25,15,29] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e2 is incomplete, acting [49,37,62,11,31] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e3 is incomplete, acting [1,49,32,56,48] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e4 is incomplete, acting [16,23,37,18,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e5 is incomplete, acting [21,38,6,23,57] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e6 is incomplete, acting [44,32,11,15,41] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e7 is incomplete, acting [35,20,42,48,26] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e8 is incomplete, acting [49,41,16,19,5] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1e9 is incomplete, acting [26,17,58,20,24] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1ea is incomplete, acting [57,23,25,26,12] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1eb is incomplete, acting [39,30,61,18,10] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1ec is incomplete, acting [21,20,11,38,4] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1ed is incomplete, acting [56,34,45,42,33] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1ee is incomplete, acting [40,53,2,27,33] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1ef is incomplete, acting [21,56,3,39,42] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f0 is incomplete, acting [32,49,45,19,2] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f1 is incomplete, acting [46,34,45,8,47] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f2 is incomplete, acting [43,39,20,30,16] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f3 is incomplete, acting [30,43,23,25,32] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f4 is incomplete, acting [30,16,29,2,8] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f5 is incomplete, acting [15,28,6,11,7] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f6 is incomplete, acting [61,25,45,34,33] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f7 is incomplete, acting [33,27,6,11,15] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f8 is incomplete, acting [47,8,30,19,7] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1f9 is incomplete, acting [11,44,58,26,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1fa is incomplete, acting [32,51,19,39,2] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1fb is incomplete, acting [14,19,61,35,30] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1fc is incomplete, acting [37,0,47,17,18] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1fd is incomplete, acting [49,20,34,62,15] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1fe is incomplete, acting [46,52,33,34,9] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
pg 10.1ff is incomplete, acting [33,21,7,19,52] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')
root@ceph-monitor02:~#
Somebody has an idea of how to fix this??
Maybe copying the data to a replicated pool with min_size=1 ?
All data are hopelessly lost?
Thanks in advance.
--
Félix Barbeira.
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com