I'm quite clear now.
> this is my test setup - that's why I'm trying to break it and fix it (best way to learn)
> this is my test setup - that's why I'm trying to break it and fix it (best way to learn)
Thanks for your feedback!!
KinjoOn Sun, Jul 5, 2015 at 8:51 PM, Jacek Jarosiewicz <jjarosiewicz@xxxxxxxxxxxxx> wrote:
Well, the docs say that when your osds get full you should add another osd - and the cluster should redistribute data by it self:
http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space
this is my test setup - that's why I'm trying to break it and fix it (best way to learn) - so I intentionally filled the cache pool and saw this problem twice - when I have full or near full osds the cluster becomes a little less responsive and I get the "blocked requests" warning and some clients hang.. I tried to delete some files manually before but I think I deleted too many or something else went wrong, because the last time I could not get the cluster working and had to reinitialize all the osds from scratch. This time it worked - but I'm worried and wondering that maybe there is a better way to go about this problem..
I was thinking that maybe - if I upped the near full and full ratio - the warning would go away and maybe I would be able to flush the cache pool. But that's only a solution for the cache pool - I'd rather not touch the normal data on the cold storage..
Cheers,
J
On 07/05/2015 01:27 PM, Shinobu Kinjo wrote:
That's good!
So was the root cause is because the osd was full? What's your thought
about that?
Was there any reason to delete any files?
Kinjo
On Sun, Jul 5, 2015 at 6:51 PM, Jacek Jarosiewicz
<jjarosiewicz@xxxxxxxxxxxxx <mailto:jjarosiewicz@xxxxxxxxxxxxx>> wrote:
ok, I got it working...
first i manually deleted some files from the full osd, set the flag
noout and restarted the osd daemon.
then i waited a while for the cluster to backfill pgs, and after
that the rados -p cache cache-try-flush-evict-all command went OK.
I'm wondering though, because this happened to me before (hanging
blocked request when osd's are near full), is there a better way
maybe to solve this problem?
J
--
Life w/ Linux <http://i-shinobu.hatenablog.com/>
--
Jacek Jarosiewicz
Administrator Systemów Informatycznych
----------------------------------------------------------------------------------------
SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie
ul. Senatorska 13/15, 00-075 Warszawa
Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego Rejestru Sądowego,
nr KRS 0000029537; kapitał zakładowy 42.756.000 zł
NIP: 957-05-49-503
Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa
----------------------------------------------------------------------------------------
SUPERMEDIA -> http://www.supermedia.pl
dostep do internetu - hosting - kolokacja - lacza - telefonia
--
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
