Hi John, 128MB is a forced minimum in the code for iser to work properly. You can ignore the shmget rdma pool failure message, It means it failed to allocate huge-pages and will revert to normal memory allocation. Can you show more from the log after it? Thanks, Roi -----Original Message----- From: John Francis [mailto:john@xxxxxxxxxxxxxxxx] Sent: Tuesday, July 23, 2013 3:17 PM To: Roi Dayan Cc: stgt@xxxxxxxxxxxxxxx Subject: Re: iser Issue, cannot allocate memory On 23 July 2013 16:59, Roi Dayan <roid@xxxxxxxxxxxx> wrote: > Hi John, > > Can you try to limit the memory pool iser is trying to allocate? > i.e.: > tgtd --iser pool_sz_mb=100 Okay, I was running an older version of tgtd that didn't support that option. Thanks to Roi's advice I've upgraded to tgtd 1.0.37 and started up the server both with the --ser pool_sz_mb=100 and without. With the limit, I get: tgtd: iser_alloc_pool(573) shmget rdma pool sz:134217728 failed Without it, I get: tgtd: iser_alloc_pool(573) shmget rdma pool sz:1073741824 failed It's funny, that it looks like the minimum it tries to allocate is 128 MB, even it I use pool_sz_mb=50, it still tries to allocate 128 MB. Could this be a kernel.shmmax or kernel.shmall issue? > > Thanks, > Roi > > > -----Original Message----- > From: stgt-owner@xxxxxxxxxxxxxxx [mailto:stgt-owner@xxxxxxxxxxxxxxx] > On Behalf Of John Francis > Sent: Tuesday, July 23, 2013 4:32 AM > To: stgt@xxxxxxxxxxxxxxx > Subject: iser Issue, cannot allocate memory > > Hi, > > I have an iser target configured. When I try to log in to that target > from another machine, it doesn't work, and the following appears in > /var/log/message: > > 10:54:33 stor1 tgtd: iser_device_init(3204) iser_init_rdma_buf_pool > failed > 10:54:37 stor1 tgtd: iser_init_rdma_buf_pool(589) ibv_reg_mr failed, Cannot allocate memory ... > > I've tried fiddling with various limits but to no avail. I'm struggling to figure out if the memory limit being hit is card related of host related. > > Using some Mellonox cards with ib_mthca module. IPoIB seems to be working fine. It should be noted that this error occurs on initial log in to the target. > > -- > Kind regards, > > John Francis > > www.recompute.com.au > 6/38 Canterbury Rd Bankstown NSW 2200 > T: (02) 9709 5775 ext 9150 > > Sign up to Hit Refresh, our newsletter with specials, discounts and tips, or follow us on Twitter @recompute_au. > -- > To unsubscribe from this list: send the line "unsubscribe stgt" in the > body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at > http://vger.kernel.org/majordomo-info.html -- Kind regards, John Francis www.recompute.com.au 6/38 Canterbury Rd Bankstown NSW 2200 T: (02) 9709 5775 ext 9150 Sign up to Hit Refresh, our newsletter with specials, discounts and tips, or follow us on Twitter @recompute_au. -- To unsubscribe from this list: send the line "unsubscribe stgt" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
