Hello Ming, Sreekanth,
I'm observing a problem where mpt3sas fails to allocate the budget_map
for any SCSI device, because attempted allocation is larger than the
maximum possible. The issue is caused by the logic used in 020b0f0a3192
("scsi: core: Replace sdev->device_busy with sbitmap")
to calculate the bitmap size. This is observed with 5.16-rc8.
The controller at hand has properties can_queue=29865 and
cmd_per_lun=7. The way these parameters are used in scsi_alloc_sdev()->
sbitmap_init_node(), this results in an sbitmap with 29865 maps, where
only a single bit is used per map. On x86_64, this results in an
attempt to allocate 29865 * 192 = 5734080 bytes for the sbitmap, which
is larger than PAGE_SIZE * (1 << (MAX_ORDER - 1)), and fails.
cmd_per_lun=7 is an extreme case, because sbitmap_calculate_shift()
returns 0 (i.e. 1 bit per word) for this case. The problem for this
controller is the very large difference between the actual
cmd_per_lun=7 and the theoretical maximum
scsi_device_max_queue_depth(sdev) = host->can_queue = 29865. The
generic sbitmap logic assumes that the initial size is at least rougly
similar to the maximum.
I believe it would make sense to limit scsi_device_max_queue_depth(),
and thus the bitmap size, to BLK_MQ_MAX_DEPTH = 10240. But even then,
we would allocate a huge bitmap for this controller - 2MiB, of which no
more than 10kiB would ever be used (and only 7 bytes (!) in the default
case).
I'm unsure how to fix this correctly. To my understanding, the reason
that the sbitmap is split over multiple cache lines is to avoid ping-
pong between CPUs accessing the bitmap simultaneously. 29865 maps is
quite obviously too much. AFAICS it makes no sense to use more than
nr_cpu_ids separate bitmap chunks, and for a single SCSI device
probably much less (like 8 or 16 chunks) would be sufficient.
That means that we'd have to calculate the "shift" argument to
sbitmap_init_node() differently.
Here's a tentative patch that would do this, please have a look.
With this patch, the driver would use a bitmap with shift 6 for my
problem case./bitma
diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index 23e1c0acdeae..3e51f8183b42 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -231,7 +231,8 @@ static void scsi_unlock_floptical(struct scsi_device *sdev,
static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
u64 lun, void *hostdata)
{
- unsigned int depth;
+ static const unsigned int BITMAP_CHUNKS = 16;
+ unsigned int depth, sb_map_chunks, sb_depth;
struct scsi_device *sdev;
struct request_queue *q;
int display_failure_msg = 1, ret;
@@ -298,17 +299,26 @@ static struct scsi_device *scsi_alloc_sdev(struct scsi_target *starget,
__scsi_init_queue(sdev->host, q);
WARN_ON_ONCE(!blk_get_queue(q));
- depth = sdev->host->cmd_per_lun ?: 1;
-
/*
* Use .can_queue as budget map's depth because we have to
* support adjusting queue depth from sysfs. Meantime use
* default device queue depth to figure out sbitmap shift
* since we use this queue depth most of times.
+ * Avoid extremely small shift if cmd_per_lun is small but
+ * greater than 3 (see sbitmap_calculate_shift()).
+ * Assume that usually no more than BITMAP_CHUNKS
+ * CPUs will access this bitmap simultaneously.
*/
+ depth = sdev->host->cmd_per_lun ?: 1;
+ sb_map_chunks = max_t(unsigned int, 1U,
+ min_t(unsigned int, nr_cpu_ids, BITMAP_CHUNKS));
+ sb_depth = max_t(unsigned int,
+ scsi_device_max_queue_depth(sdev) / sb_map_chunks,
+ depth);
+
if (sbitmap_init_node(&sdev->budget_map,
scsi_device_max_queue_depth(sdev),
- sbitmap_calculate_shift(depth),
+ sbitmap_calculate_shift(sb_depth),
GFP_KERNEL, sdev->request_queue->node,
false, true)) {
put_device(&starget->dev);
Regards,
Martin
Below is a dmesg snippet illustrating the problem.
See "Current Controller Queue Depth" (=can_queue).
[ 2760.377997] mpt2sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(9), sge_per_io(128), chains_per_io(15)
[ 2760.475225] mpt2sas_cm0: request pool(0x000000004d742465) - dma(0x447800000): depth(30127), frame_size(128), pool_size(3765 kB)
[ 2761.787680] mpt2sas_cm0: sense pool(0x0000000069af807c) - dma(0x447400000): depth(29868), element_size(96), pool_size (2800 kB)
[ 2761.898550] mpt2sas_cm0: sense pool(0x0000000069af807c)- dma(0x447400000): depth(29868),element_size(96), pool_size(0 kB)
[ 2762.002396] mpt2sas_cm0: reply pool(0x000000005def89ce) - dma(0x44b400000): depth(30191), frame_size(128), pool_size(3773 kB)
[ 2762.107551] mpt2sas_cm0: config page(0x00000000dc01e404) - dma(0x44d69f000): size(512)
[ 2762.183235] mpt2sas_cm0: Allocated physical memory: size(66932 kB)
[ 2762.242790] mpt2sas_cm0: Current Controller Queue Depth(29865),Max Controller Queue Depth(32455)
[ 2762.326393] mpt2sas_cm0: Scatter Gather Elements per IO(128)
[ 2762.426639] mpt2sas_cm0: LSISAS2116: FWVersion(15.00.00.00), ChipRevision(0x02), BiosVersion(07.29.00.00)
[ 2762.517588] mpt2sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[ 2762.626129] scsi host1: Fusion MPT SAS Host
[ 2762.669095] blk-mq: reduced tag depth to 10240
[ 2762.721135] mpt2sas_cm0: sending port enable !!
[ 2762.768419] mpt2sas_cm0: hba_port entry: 00000000c01410d1, port: 255 is added to hba_port list
[ 2762.858130] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b002d80ae0), phys(16)
[ 2762.937991] mpt2sas_cm0: handle(0x11) sas_address(0x500a0b838933700c) port_type(0x4)
[ 2763.012583] mpt2sas_cm0: handle(0x14) sas_address(0x50080e51beede010) port_type(0x4)
[ 2763.086293] mpt2sas_cm0: handle(0x12) sas_address(0x500a0b83763eb00c) port_type(0x4)
[ 2763.161090] mpt2sas_cm0: handle(0x13) sas_address(0x50080e51bf1f0010) port_type(0x4)
[ 2763.243850] mpt2sas_cm0: port enable: SUCCESS
[ 2763.290141] ------------[ cut here ]------------
[ 2763.336935] WARNING: CPU: 15 PID: 9641 at mm/page_alloc.c:5344 __alloc_pages+0x2a1/0x340
[ 2764.707134] CPU: 15 PID: 9641 Comm: kworker/u258:2 Tainted: G I E 5.16.0-rc8-mw #1 openSUSE Tumbleweed (unreleased) d2750100410fdc3981005f9733dae65153e2c621
[ 2764.707140] Hardware name: HP ProLiant DL560 Gen8, BIOS P77
05/24/2019
[ 2764.707144] Workqueue: events_unbound async_run_entry_fn
[ 2764.707153] RIP: 0010:__alloc_pages+0x2a1/0x340
[ 2764.707155] Code: 3d fb f6 a6 01 01 76 0e 48 83 b8 d0 f9 ff ff 00 0f 84 e2 fe ff ff 80 ce 01 e9 da fe ff ff 81 e7 00 20 00 00 0f 85 79 ff ff ff <0f> 0b e9 72 ff ff ff 48 89 c7 e8 30 c8 fb ff e9 91 fe ff ff 31 c0
[ 2764.707158] RSP: 0018:ffffb5dfc688fa80 EFLAGS: 00010246
[ 2764.707160] RAX: 0000000000000000 RBX: 00000000000074a9 RCX: 0000000000000000
[ 2764.707161] RDX: 0000000000000001 RSI: 000000000000000b RDI: 0000000000000000
[ 2764.707163] RBP: 000000000000000b R08: 0000000000000000 R09: 0000000000000000
[ 2764.707164] R10: 000000000000007f R11: 000000000000000f R12: 0000000000577ec0
[ 2764.707166] R13: 0000000000000cc0 R14: 00000000ffffffff R15: 0000000000000080
[ 2764.707167] FS: 0000000000000000(0000) GS:ffff8954ef7c0000(0000) knlGS:0000000000000000
[ 2764.707169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2764.707170] CR2: 00007fdb29c3ffa0 CR3: 00000002c6a10006 CR4: 00000000000606e0
[ 2764.707172] Call Trace:
[ 2764.707178] <TASK>
[ 2764.707185] kmalloc_large_node+0x3a/0xa0
[ 2764.707192] __kmalloc_node+0x2b7/0x330
[ 2764.707195] sbitmap_init_node+0x7d/0x1a0
[ 2764.707204] scsi_alloc_sdev+0x25e/0x330
[ 2764.707210] scsi_probe_and_add_lun+0x43d/0xdf0
[ 2764.707214] __scsi_scan_target+0xf9/0x610
...
[ 2766.769381] </TASK>
[ 2766.769382] ---[ end trace ac1aec9da4689c68 ]---
[ 2766.769387] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
