Sorry, it's been very busy. The next step would to try to get a heap dump. You can start a heap profile on osd N by: ceph osd tell N heap start_profiler and you can get it to dump the collected profile using ceph osd tell N heap dump. The dumps should show up in the osd log directory. Assuming the heap profiler is working correctly, you can look at the dump using pprof in google-perftools. On Wed, Dec 19, 2012 at 8:37 AM, Sébastien Han <han.sebastien@xxxxxxxxx> wrote: > No more suggestions? :( > -- > Regards, > Sébastien Han. > > > On Tue, Dec 18, 2012 at 6:21 PM, Sébastien Han <han.sebastien@xxxxxxxxx> wrote: >> Nothing terrific... >> >> Kernel logs from my clients are full of "libceph: osd4 >> 172.20.11.32:6801 socket closed" >> >> I saw this somewhere on the tracker. >> >> Does this harm? >> >> Thanks. >> >> -- >> Regards, >> Sébastien Han. >> >> >> >> On Mon, Dec 17, 2012 at 11:55 PM, Samuel Just <sam.just@xxxxxxxxxxx> wrote: >>> >>> What is the workload like? >>> -Sam >>> >>> On Mon, Dec 17, 2012 at 2:41 PM, Sébastien Han <han.sebastien@xxxxxxxxx> wrote: >>> > Hi, >>> > >>> > No, I don't see nothing abnormal in the network stats. I don't see >>> > anything in the logs... :( >>> > The weird thing is that one node over 4 seems to take way more memory >>> > than the others... >>> > >>> > -- >>> > Regards, >>> > Sébastien Han. >>> > >>> > >>> > On Mon, Dec 17, 2012 at 11:31 PM, Sébastien Han <han.sebastien@xxxxxxxxx> wrote: >>> >> >>> >> Hi, >>> >> >>> >> No, I don't see nothing abnormal in the network stats. I don't see anything in the logs... :( >>> >> The weird thing is that one node over 4 seems to take way more memory than the others... >>> >> >>> >> -- >>> >> Regards, >>> >> Sébastien Han. >>> >> >>> >> >>> >> >>> >> On Mon, Dec 17, 2012 at 7:12 PM, Samuel Just <sam.just@xxxxxxxxxxx> wrote: >>> >>> >>> >>> Are you having network hiccups? There was a bug noticed recently that >>> >>> could cause a memory leak if nodes are being marked up and down. >>> >>> -Sam >>> >>> >>> >>> On Mon, Dec 17, 2012 at 12:28 AM, Sébastien Han <han.sebastien@xxxxxxxxx> wrote: >>> >>> > Hi guys, >>> >>> > >>> >>> > Today looking at my graphs I noticed that one over 4 ceph nodes used a >>> >>> > lot of memory. It keeps growing and growing. >>> >>> > See the graph attached to this mail. >>> >>> > I run 0.48.2 on Ubuntu 12.04. >>> >>> > >>> >>> > The other nodes also grow, but slowly than the first one. >>> >>> > >>> >>> > I'm not quite sure about the information that I have to provide. So >>> >>> > let me know. The only thing I can say is that the load haven't >>> >>> > increase that much this week. It seems to be consuming and not giving >>> >>> > back the memory. >>> >>> > >>> >>> > Thank you in advance. >>> >>> > >>> >>> > -- >>> >>> > Regards, >>> >>> > Sébastien Han. >>> >> >>> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
