How to reduce min_size of an EC pool?

[Félix Barbeira <fbarbeira@xxxxxxxxx>]

I want to bring back my cluster to HEALTHY state because right now I have not access to the data. 

I have an 3+2 EC pool on a 5 node cluster. 3 nodes were lost, all data wiped. They were reinstalled and added to cluster again.

The "ceph health detail" command says to reduce min_size number to a value lower than 3, but:

root@ceph-monitor02:~# ceph osd pool set default.rgw.buckets.data min_size 2

Error EINVAL: pool min_size must be between 3 and 5

root@ceph-monitor02:~#

This is the situation:

root@ceph-monitor01:~# ceph -s

  cluster:

    id:     ce78b02d-03df-4f9e-a35a-31b5f05c4c63

    health: HEALTH_WARN

            Reduced data availability: 515 pgs inactive, 512 pgs incomplete

  services:

    mon: 3 daemons, quorum ceph-monitor01,ceph-monitor03,ceph-monitor02

    mgr: ceph-monitor02(active), standbys: ceph-monitor01, ceph-monitor03

    osd: 57 osds: 57 up, 57 in

  data:

    pools:   8 pools, 568 pgs

    objects: 4.48 M objects, 10 TiB

    usage:   24 TiB used, 395 TiB / 419 TiB avail

    pgs:     0.528% pgs unknown

             90.141% pgs not active

             512 incomplete

             53  active+clean

             3   unknown

root@ceph-monitor01:~#

And this is the output of health detail:

root@ceph-monitor01:~# ceph health detail

HEALTH_WARN Reduced data availability: 515 pgs inactive, 512 pgs incomplete

PG_AVAILABILITY Reduced data availability: 515 pgs inactive, 512 pgs incomplete

    pg 10.1cd is stuck inactive since forever, current state incomplete, last acting [9,48,41,58,17] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1ce is incomplete, acting [3,13,14,42,21] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1cf is incomplete, acting [36,27,3,39,51] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d0 is incomplete, acting [29,9,38,4,56] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d1 is incomplete, acting [2,34,17,7,30] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d2 is incomplete, acting [41,45,53,13,32] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d3 is incomplete, acting [7,28,15,20,3] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d4 is incomplete, acting [11,40,25,23,0] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d5 is incomplete, acting [32,51,20,57,28] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d6 is incomplete, acting [2,53,8,16,15] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d7 is incomplete, acting [1,2,33,43,42] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d8 is incomplete, acting [27,49,9,48,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1d9 is incomplete, acting [37,8,7,11,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1da is incomplete, acting [27,14,33,15,53] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1db is incomplete, acting [58,53,6,26,4] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1dc is incomplete, acting [21,12,47,35,19] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1dd is incomplete, acting [51,4,52,24,7] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1de is incomplete, acting [38,29,21,41,44] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1df is incomplete, acting [25,4,30,61,11] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e0 is incomplete, acting [27,57,21,6,13] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e1 is incomplete, acting [8,7,25,15,29] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e2 is incomplete, acting [49,37,62,11,31] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e3 is incomplete, acting [1,49,32,56,48] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e4 is incomplete, acting [16,23,37,18,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e5 is incomplete, acting [21,38,6,23,57] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e6 is incomplete, acting [44,32,11,15,41] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e7 is incomplete, acting [35,20,42,48,26] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e8 is incomplete, acting [49,41,16,19,5] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1e9 is incomplete, acting [26,17,58,20,24] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1ea is incomplete, acting [57,23,25,26,12] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1eb is incomplete, acting [39,30,61,18,10] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1ec is incomplete, acting [21,20,11,38,4] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1ed is incomplete, acting [56,34,45,42,33] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1ee is incomplete, acting [40,53,2,27,33] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1ef is incomplete, acting [21,56,3,39,42] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f0 is incomplete, acting [32,49,45,19,2] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f1 is incomplete, acting [46,34,45,8,47] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f2 is incomplete, acting [43,39,20,30,16] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f3 is incomplete, acting [30,43,23,25,32] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f4 is incomplete, acting [30,16,29,2,8] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f5 is incomplete, acting [15,28,6,11,7] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f6 is incomplete, acting [61,25,45,34,33] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f7 is incomplete, acting [33,27,6,11,15] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f8 is incomplete, acting [47,8,30,19,7] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1f9 is incomplete, acting [11,44,58,26,20] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1fa is incomplete, acting [32,51,19,39,2] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1fb is incomplete, acting [14,19,61,35,30] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1fc is incomplete, acting [37,0,47,17,18] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1fd is incomplete, acting [49,20,34,62,15] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1fe is incomplete, acting [46,52,33,34,9] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

    pg 10.1ff is incomplete, acting [33,21,7,19,52] (reducing pool default.rgw.buckets.data min_size from 3 may help; search ceph.com/docs for 'incomplete')

root@ceph-monitor02:~#

Somebody has an idea of how to fix this??

Maybe copying the data to a replicated pool with min_size=1 ?

All data are hopelessly lost?

Thanks in advance.

--

Félix Barbeira.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com