Re: Recovering full OSD

[Eric Eastman <eric.eastman@xxxxxxxxxxxxxx>]

Under Jewel 10.2.2 I have also had to delete PG directories to get very full OSDs to restart. I first use "du -sh *" under the "current" directory to find which OSD directories are the fullest on the full OSD disk, and pick 1 of the fullest.  I then look at the PG map and verify the PG is replicated and is on another running OSD.  I also reweight the full OSD before deleting the selected PG tree on the full OSD, using:

rm -rf PG_head 

  where PG is the PG number

After the directory is removed, I restart the OSD.  Once the OSD is back up running and has some free space, I do a deep-scrub on the PG.  So far I have had no issues using this procedure.

Eric

On Mon, Aug 8, 2016 at 7:32 AM, Gerd Jakobovitsch <gerd@xxxxxxxxxxxxx> wrote:

I got to this situation several times, due to a strange behavior in the xfs filesystem - I initially ran on debian, afterwards reinstalled the nodes to centos7, kernel 3.10.0-229.14.1.el7.x86_64, package xfsprogs-3.2.1-6.el7.x86_64. Around 75-80% of usage shown with df, the disk is already full.

To delete PGs in order to restart the OSD, I first lowered the weight of the affected OSD, and observed which PGs started backfilling elsewhere. Then I deleted some of these backfilling PGs before trying to restart the OSD. It worked without data loss.

Em 08-08-2016 08:19, Mykola Dvornik escreveu:

@Shinobu

According to

http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/

"If you cannot start an OSD because it is full, you may delete some data by deleting some placement group directories in the full OSD."

On 8 August 2016 at 13:16, Shinobu Kinjo <shinobu.kj@xxxxxxxxx> wrote:

On Mon, Aug 8, 2016 at 8:01 PM, Mykola Dvornik <mykola.dvornik@xxxxxxxxx> wrote:
> Dear ceph community,
>
> One of the OSDs in my cluster cannot start due to the
>
> ERROR: osd init failed: (28) No space left on device
>
> A while ago it was recommended to manually delete PGs on the OSD to let it
> start.

Who recommended that?

>
> So I am wondering was is the recommended way to fix this issue for the
> cluster running Jewel release (10.2.2)?
>
> Regards,
>
> --
>  Mykola
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



--
Email:
shinobu@xxxxxxxxx
shinobu@xxxxxxxxxx

--

 Mykola 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

--

--

	Gerd Jakobovitsch Diretoria de Tecnologia +55 11 3030-3456 Avalie a Mandic Cloud | Como está sua satisfação? Vendas: 4007-2442 Suporte 24h: 4007-1858 | 400-365-24 Mandic. Somos Especialistas em Cloud.

 

--

As informações contidas nesta mensagem são CONFIDENCIAIS, protegidas pelo sigilo legal e por direitos autorais. A divulgação, distribuição, reprodução ou qualquer forma de utilização do teor deste documento depende de autorização do emissor, sujeitando-se o infrator às sanções legais. Caso esta comunicação tenha sido recebida por engano, favor avisar imediatamente, respondendo esta mensagem.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com