Re: Help Basically..

David Turner <drakonstein@xxxxxxxxx> · Sun, 2 Sep 2018 10:15:37 -0400

When the first node went offline with a dead SSD journal, all of the dates on the OSDs was useless. Unless you could flush the journals, you can't guarantee that a wire the cluster think happened actually made it to the disk.  The proper procedure here is to remove those OSDs and add them again as new OSDs.
`ceph health detail` will give you some more information on the blocked requests. Depending on what that shows you can often find the OSD that is causing the problems.  But your biggest problem is that you have dishes with potentially inconsistent data in your closer.

On Sun, Sep 2, 2018, 4:42 AM Lee <lquince@xxxxxxxxx> wrote:
Running 0.94.5 as part of a Openstack enviroment, our ceph setup is 3x OSD Nodes 3x MON Nodes, yesterday we had a aircon outage in our hosting enviroment, 1 OSD node failed (offline with a the journal SSD dead) left with 2 nodes running correctly, 2 hours later a second OSD node failed complaining of readwrite errors to the physical drives, i assume this was a heat issue as when rebooted this came back online ok and ceph started to repair itself. We have since brought the first failed node back on by replacing the ssd and recreating the journals hoping it would all repair.. Our pools are min 2 repl. 

The problem we have is client IO (read) is totally blocked, and when I query the stuck PG's it just hangs..

For example the check version command just errors with:

Error EINTR: problem getting command descriptions from on various OSD's so I cannot even query the inactive PG's

root@node31-a4:~# ceph -s
    cluster 7c24e1b9-24b3-4a1b-8889-9b2d7fd88cd2
     health HEALTH_WARN
            83 pgs backfill
            2 pgs backfill_toofull
            3 pgs backfilling
            48 pgs degraded
            1 pgs down
            31 pgs incomplete
            1 pgs recovering
            29 pgs recovery_wait
            1 pgs stale
            48 pgs stuck degraded
            31 pgs stuck inactive
            1 pgs stuck stale
            148 pgs stuck unclean
            17 pgs stuck undersized
            17 pgs undersized
            599 requests are blocked > 32 sec
            recovery 111489/4697618 objects degraded (2.373%)
            recovery 772268/4697618 objects misplaced (16.440%)
            recovery 1/2171314 unfound (0.000%)
     monmap e5: 3 mons at {bc07s12-a7=172.27.16.11:6789/0,bc07s13-a7=172.27.16.21:6789/0,bc07s14-a7=172.27.16.15:6789/0}
            election epoch 198, quorum 0,1,2 bc07s12-a7,bc07s14-a7,bc07s13-a7
     osdmap e18727: 25 osds: 25 up, 25 in; 90 remapped pgs
      pgmap v70996322: 1792 pgs, 13 pools, 8210 GB data, 2120 kobjects
            16783 GB used, 6487 GB / 23270 GB avail
            111489/4697618 objects degraded (2.373%)
            772268/4697618 objects misplaced (16.440%)
            1/2171314 unfound (0.000%)
                1639 active+clean
                  66 active+remapped+wait_backfill
                  30 incomplete
                  25 active+recovery_wait+degraded
                  15 active+undersized+degraded+remapped+wait_backfill
                   4 active+recovery_wait+degraded+remapped
                   4 active+clean+scrubbing
                   2 active+remapped+wait_backfill+backfill_toofull
                   1 down+incomplete
                   1 active+remapped+backfilling
                   1 active+clean+scrubbing+deep
                   1 stale+active+undersized+degraded
                   1 active+undersized+degraded+remapped+backfilling
                   1 active+degraded+remapped+backfilling
                   1 active+recovering+degraded
recovery io 29385 kB/s, 7 objects/s
  client io 5877 B/s wr, 1 op/s

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com