Dave, I share you my little script for now if you want it:
#!/bin/bash
for i in $(ps aux | grep [c]eph-osd | awk '{print $4}')
do
MEM_INTEGER=$(echo $i | cut -d '.' -f1)
OSD=$(ps aux | grep [c]eph-osd | grep "$i " | awk '{print $13}')
if [[ $MEM_INTEGER -ge 25 ]];then
service ceph restart osd.$OSD >> /dev/null
if [ $? -eq 0 ]; then
logger -t ceph-memory-usage "The OSD number $OSD has been restarted
since it was using $i % of the memory"
else
logger -t ceph-memory-usage "ERROR while
restarting the OSD daemon"
fi
else
logger -t ceph-memory-usage "The OSD number $OSD is
only using $i % of the memory, doing nothing"
fi
logger -t ceph-memory-usage "Waiting 60 seconds before testing the next OSD..."
sleep 60
done
logger -t ceph-memory-usage "Ceph state after memory check operation
is: $(ceph health)"
Crons run with 10 min interval everyday for each storage node ;-).
Waiting for some Inktank guys now :-).
--
Regards,
Sébastien Han.
On Wed, Jan 9, 2013 at 10:42 PM, Dave Spano <dspano@xxxxxxxxxxxxxx> wrote:
> That's very good to know. I'll be restarting ceph-osd right now! Thanks for the heads up!
>
> Dave Spano
> Optogenics
> Systems Administrator
>
>
>
> ----- Original Message -----
>
> From: "Sébastien Han" <han.sebastien@xxxxxxxxx>
> To: "Dave Spano" <dspano@xxxxxxxxxxxxxx>
> Cc: "ceph-devel" <ceph-devel@xxxxxxxxxxxxxxx>, "Samuel Just" <sam.just@xxxxxxxxxxx>
> Sent: Wednesday, January 9, 2013 11:35:13 AM
> Subject: Re: OSD memory leaks?
>
> If you wait too long, the system will trigger OOM killer :D, I already
> experienced that unfortunately...
>
> Sam?
>
> On Wed, Jan 9, 2013 at 5:10 PM, Dave Spano <dspano@xxxxxxxxxxxxxx> wrote:
>> OOM killer
>
>
>
> --
> Regards,
> Sébastien Han.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
