> > > On Sat, Sep 14, 2019 at 1:46 AM Bart Van Assche <bvanassche@xxxxxxx> wrote: > > >> A more general question is why ibnbd needs its own queue management > > >> while no other block driver needs this? > > > > > > Each IBNBD device promises to have a queue_depth (of say 512) on each > > > of its num_cpus hardware queues. In fact we can only process a > > > queue_depth inflights at once on the whole ibtrs session connecting a > > > given client with a given server. Those 512 inflights (corresponding > > > to the number of buffers reserved by the server for this particular > > > client) have to be shared among all the devices mapped on this > > > session. This leads to the situation, that we receive more requests > > > than we can process at the moment. So we need to stop queues and start > > > them again later in some fair fashion. > > > > Can a single CPU really sustain a queue depth of 512 commands? Is it > > really necessary to have one hardware queue per CPU or is e.g. four > > queues per NUMA node sufficient? Has it been considered to send the > > number of hardware queues that the initiator wants to use and also the > > command depth per queue during login to the target side? That would > > allow the target side to allocate an independent set of buffers for each > > initiator hardware queue and would allow to remove the queue management > > at the initiator side. This might even yield better performance. > We needed a way which would allow us to address one particular > requirement: we'd like to be able to "enforce" that a response to an > IO would be processed on the same CPU the IO was originally submitted > on. In order to be able to do so we establish one rdma connection per > cpu, each having a separate cq_vector. The administrator can then > assign the corresponding IRQs to distinct CPUs. The server always > replies to an IO on the same connection he received the request on. If > the administrator did configure the /proc/irq/y/smp_affinity > accordingly, the response sent by the server will generate interrupt > on the same cpu, the IO was originally submitted on. The administrator > can configure IRQs differently, for example assign a given irq > (<->cq_vector) to a range of cpus belonging to a numa node, or > whatever assignment is best for his use-case. > Our transport module IBTRS establishes number of cpus connections > between a client and a server. The user of the transport module (i.e. > IBNBD) has no knowledge about the rdma connections, it only has a > pointer to an abstract "session", which connects him somehow to a > remote host. IBNBD as a user of IBTRS creates block devices and uses a > given "session" to send IOs from all the block devices it created for > that session. That means IBNBD is limited in maximum number of his > inflights toward a given remote host by the capability of the > corresponding "session". So it needs to share the resources provided > by the session (in our current model those resources are in fact some > pre registered buffers on server side) among his devices. > It is possible to extend the IBTRS API so that the user (IBNBD) could > specify how many connections he wants to have on the session to be > established. It is also possible to extend the ibtrs_clt_get_tag API > (this is to get a send "permit") with a parameter specifying the > connection, the future IO is to be send on. > We now might have to change our communication model in IBTRS a bit in > order to fix the potential security problem raised during the recent > RDMA MC: https://etherpad.net/p/LPC2019_RDMA. > I'm not familiar with dm code, but don't they need to deal with the same situation: if I configure 100 logical volumes on top of a single NVME drive with X hardware queues, each queue_depth deep, then each dm block device would need to advertise X hardware queues in order to achieve highest performance in case only this one volume is accessed, while in fact those X physical queues have to be shared among all 100 logical volumes, if they are accessed in parallel?
