Re: best way to resolve 'stale+active+clean' after disk failure

Ben Hines <bhines@xxxxxxxxx> · Thu, 6 Apr 2017 16:04:18 -0700

Personally before extreme measures like marking lost,  i would try bringing up the osd, so it's up and out -- i believe the data will still be found and re balanced away from it by Ceph.
-Ben

On Thu, Apr 6, 2017 at 11:20 AM, David Welch <dwelch@xxxxxxxxxxxx> wrote:

    Hi,

      We had a disk on the cluster that was not responding properly and
      causing 'slow requests'. The osd on the disk was stopped and the
      osd was marked down and then out. Rebalancing succeeded but
      (some?) pgs from that osd are now stuck in stale+active+clean
      state, which is not being resolved (see below for query results).

    My question: is it better to mark this osd as "lost" (i.e. 'ceph
      osd lost 14') or to remove the osd as detailed here:

      https://www.sebastien-han.fr/blog/2015/12/11/ceph-properly-remove-an-osd/

    Thanks,

      David

        $ ceph health detail

        HEALTH_ERR 17 pgs are stuck inactive for more than 300 seconds;
        17 pgs stale; 17 pgs stuck stale

        pg 7.f3 is stuck stale for 6138.330316, current state
        stale+active+clean, last acting [14]

        pg 7.bd is stuck stale for 6138.330365, current state
        stale+active+clean, last acting [14]

        pg 7.b6 is stuck stale for 6138.330374, current state
        stale+active+clean, last acting [14]

        pg 7.c5 is stuck stale for 6138.330363, current state
        stale+active+clean, last acting [14]

        pg 7.ac is stuck stale for 6138.330385, current state
        stale+active+clean, last acting [14]

        pg 7.5b is stuck stale for 6138.330678, current state
        stale+active+clean, last acting [14]

        pg 7.1b4 is stuck stale for 6138.330409, current state
        stale+active+clean, last acting [14]

        pg 7.182 is stuck stale for 6138.330445, current state
        stale+active+clean, last acting [14]

        pg 7.1f8 is stuck stale for 6138.330720, current state
        stale+active+clean, last acting [14]

        pg 7.53 is stuck stale for 6138.330697, current state
        stale+active+clean, last acting [14]

        pg 7.1d2 is stuck stale for 6138.330663, current state
        stale+active+clean, last acting [14]

        pg 7.70 is stuck stale for 6138.330742, current state
        stale+active+clean, last acting [14]

        pg 7.14f is stuck stale for 6138.330585, current state
        stale+active+clean, last acting [14]

        pg 7.23 is stuck stale for 6138.330610, current state
        stale+active+clean, last acting [14]

        pg 7.153 is stuck stale for 6138.330600, current state
        stale+active+clean, last acting [14]

        pg 7.cc is stuck stale for 6138.330409, current state
        stale+active+clean, last acting [14]

        pg 7.16b is stuck stale for 6138.330509, current state
        stale+active+clean, last acting [14]

        $ ceph pg dump_stuck stale

      ok

      pg_stat    state    up    up_primary    acting   
        acting_primary

      7.f3    stale+active+clean    [14]    14    [14]    14

      7.bd    stale+active+clean    [14]    14    [14]    14

      7.b6    stale+active+clean    [14]    14    [14]    14

      7.c5    stale+active+clean    [14]    14    [14]    14

      7.ac    stale+active+clean    [14]    14    [14]    14

      7.5b    stale+active+clean    [14]    14    [14]    14

      7.1b4    stale+active+clean    [14]    14    [14]    14

      7.182    stale+active+clean    [14]    14    [14]    14

      7.1f8    stale+active+clean    [14]    14    [14]    14

      7.53    stale+active+clean    [14]    14    [14]    14

      7.1d2    stale+active+clean    [14]    14    [14]    14

      7.70    stale+active+clean    [14]    14    [14]    14

      7.14f    stale+active+clean    [14]    14    [14]    14

      7.23    stale+active+clean    [14]    14    [14]    14

      7.153    stale+active+clean    [14]    14    [14]    14

      7.cc    stale+active+clean    [14]    14    [14]    14

      7.16b    stale+active+clean    [14]    14    [14]    14

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com