Re: problem with cache tier

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I'm quite clear now.

> this is my test setup - that's why I'm trying to break it and fix it (best way to learn)

Thanks for your feedback!!

 Kinjo


On Sun, Jul 5, 2015 at 8:51 PM, Jacek Jarosiewicz <jjarosiewicz@xxxxxxxxxxxxx> wrote:
Well, the docs say that when your osds get full you should add another osd - and the cluster should redistribute data by it self:

http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space

this is my test setup - that's why I'm trying to break it and fix it (best way to learn) - so I intentionally filled the cache pool and saw this problem twice - when I have full or near full osds the cluster becomes a little less responsive and I get the "blocked requests" warning and some clients hang.. I tried to delete some files manually before but I think I deleted too many or something else went wrong, because the last time I could not get the cluster working and had to reinitialize all the osds from scratch. This time it worked - but I'm worried and wondering that maybe there is a better way to go about this problem..

I was thinking that maybe - if I upped the near full and full ratio - the warning would go away and maybe I would be able to flush the cache pool. But that's only a solution for the cache pool - I'd rather not touch the normal data on the cold storage..

Cheers,
J

On 07/05/2015 01:27 PM, Shinobu Kinjo wrote:
That's good!

So was the root cause is because the osd was full? What's your thought
about that?
Was there any reason to delete any files?

  Kinjo


On Sun, Jul 5, 2015 at 6:51 PM, Jacek Jarosiewicz
<jjarosiewicz@xxxxxxxxxxxxx <mailto:jjarosiewicz@xxxxxxxxxxxxx>> wrote:

    ok, I got it working...

    first i manually deleted some files from the full osd, set the flag
    noout and restarted the osd daemon.

    then i waited a while for the cluster to backfill pgs, and after
    that the rados -p cache cache-try-flush-evict-all command went OK.

    I'm wondering though, because this happened to me before (hanging
    blocked request when osd's are near full), is there a better way
    maybe to solve this problem?

    J

--
Life w/ Linux <http://i-shinobu.hatenablog.com/>


--
Jacek Jarosiewicz
Administrator Systemów Informatycznych

----------------------------------------------------------------------------------------
SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie
ul. Senatorska 13/15, 00-075 Warszawa
Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego Rejestru Sądowego,
nr KRS 0000029537; kapitał zakładowy 42.756.000 zł
NIP: 957-05-49-503
Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa

----------------------------------------------------------------------------------------
SUPERMEDIA ->   http://www.supermedia.pl
dostep do internetu - hosting - kolokacja - lacza - telefonia



--
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux