Greg, Updating to the new kernel updating the gcc version too. Recent kernel is changing tcmalloc version too, but, 3.16 has old tcmalloc but still exhibiting the issue. Yes, the behavior is very confusing and compiler is main variable I could think of from application perspective. If you have a 3.16/3.19 kernel, you could reproduce this following these steps. 1. Build ceph-hammer code base 2. Run with single OSD. 3. Create an image and run a fio-bed workload from client (say 16K bs, 8 num_jobs) 4. run 'dstat -m' and observe the memory usage. What I am thinking of doing is to install ceph from ceph.com and see the behavior. Thanks & Regards Somnath -----Original Message----- From: Gregory Farnum [mailto:greg@xxxxxxxxxxx] Sent: Monday, June 29, 2015 3:53 AM To: Somnath Roy Cc: ceph-devel@xxxxxxxxxxxxxxx Subject: Re: Probable memory leak in Hammer write path ? I'm confused. Changing the kernel you're using is changing the apparent memory usage of a userspace application (Ceph)? Are you changing the compiler when you change kernel versions? -Greg On Mon, Jun 29, 2015 at 3:35 AM, Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: > Some more data point.. > > 1. I am not seeing this in 3.13.0-24-generic > > 2. Seeing this in 3.16.0-23-generic , 3.19.0-21-generic > > Could this be related to gcc 4.9.* ? > > Thanks & Regards > Somnath > > -----Original Message----- > From: Somnath Roy > Sent: Saturday, June 27, 2015 5:57 PM > To: ceph-devel@xxxxxxxxxxxxxxx > Subject: Probable memory leak in Hammer write path ? > > Hi, > I am chasing a substantial memory leak in latest Hammer code base in the write path since yesterday and wanted to know if anybody else is also observing this or not. This is as simple as running a fio-rbd random_write workload in my single OSD server with say block size 16K and num_jobs = 8. I am seeing memory is increasing very substantially and eventually OSD stopped responding. > I tried it on two different servers (just to make sure since I was playing with lot of kernel param of late), but results are similar. > Digging down the code and short circuiting different layers, I got the following. > > 1. Seeing the nature of leak, it seems the entire transaction is leaking. > > 2. Code is deleting the transaction with a help of C_DeleteTransaction context and commenting out the following line and deleting the transaction (op_t) from submit_transaction(), seems resolved the leak. > > /*op_t->register_on_applied( > new ObjectStore::C_DeleteTransaction(op_t));*/ > > > 3. In my case, I short circuited the queue_transaction and that's why it is working, but in reality, we can't delete the transaction from submit_transaction(). Code seems to be doing proper way , but I am not able to find out yet why it is leaking memory during deleting it async way. > > Appreciate if anybody try out latest hammer building from source and confirm the behavior. > > Thanks & Regards > Somnath > > ________________________________ > > PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > info at http://vger.kernel.org/majordomo-info.html ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
