It crashes all the same
New backtrace attached - looks very similar to the old one, although
not identical.
On Mon, Feb 10, 2020 at 10:25 PM Jens Axboe <axboe@xxxxxxxxx> wrote:
>
> On 2/10/20 6:22 PM, Glauber Costa wrote:
> > Hello my dear io_uring friends
> >
> > Today I tried to run my tests on a different, more powerful hardware
> > (70+ Hyperthreads) and it crashed on creating the ring.
> >
> > I don't recall anything fancy in my code for creating the ring -
> > except maybe that I size the cq ring differently than the sq ring.
> >
> > The backtrace attached leads me to believe that we just accessed a
> > null struct somewhere
>
> Yeah, but it's in the alloc code, it's not in io-wq/io_uring.
> Here's where it is crashing:
>
> struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
> {
> [...]
> for_each_node(node) {
> struct io_wqe *wqe;
>
> wqe = kzalloc_node(sizeof(struct io_wqe), GFP_KERNEL, node);
>
> I guess the node isn't online, and that's why it's crashing. Try the
> below for starters, it should get it going.
>
>
> diff --git a/fs/io-wq.c b/fs/io-wq.c
> index 182aa17dc2ca..3898165baccb 100644
> --- a/fs/io-wq.c
> +++ b/fs/io-wq.c
> @@ -1115,8 +1116,11 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
>
> for_each_node(node) {
> struct io_wqe *wqe;
> + int alloc_node = node;
>
> - wqe = kzalloc_node(sizeof(struct io_wqe), GFP_KERNEL, node);
> + if (!node_online(alloc_node))
> + alloc_node = NUMA_NO_NODE;
> + wqe = kzalloc_node(sizeof(struct io_wqe), GFP_KERNEL, alloc_node);
> if (!wqe)
> goto err;
> wq->wqes[node] = wqe;
>
> --
> Jens Axboe
>
[ 52.976723] Oops: 0000 [#1] SMP NOPTI
[ 52.976763] CPU: 56 PID: 2107 Comm: io_wq_manager Not tainted 5.5.0+ #7
[ 52.976826] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
[ 52.976923] RIP: 0010:__alloc_pages_nodemask+0x132/0x340
[ 52.976975] Code: 18 01 75 04 41 80 ce 80 89 e8 48 8b 54 24 08 8b 74 24 1c c1 e8 0c 48 8b 3c 24 83 e0 01 88 44 24 20 48 85 d2 0f 85 74 01 00 00 <3b> 77 08 0f 82 6b 01 00 00 48 89 7c 24 10 89 ea 48 8b 07 b9 00 02
[ 52.977144] RSP: 0018:ffffa48ece75bc88 EFLAGS: 00010246
[ 52.977198] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000e8e8
[ 52.977265] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000002080
[ 52.977332] RBP: 0000000000012cc0 R08: ffff8b54d83a8a00 R09: 0000000000000002
[ 52.977399] R10: 0000000000000dc0 R11: ffff8b54e0800100 R12: 0000000000000000
[ 52.977465] R13: 0000000000012cc0 R14: 0000000000000001 R15: ffff8b54e11300f0
[ 52.977530] FS: 0000000000000000(0000) GS:ffff8b54e1100000(0000) knlGS:0000000000000000
[ 52.977605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 52.977660] CR2: 0000000000002088 CR3: 000000d7e660a004 CR4: 00000000007606e0
[ 52.977725] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 52.977791] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 52.977858] PKRU: 55555554
[ 52.977887] Call Trace:
[ 52.977926] alloc_slab_page+0x46/0x320
[ 52.977967] new_slab+0x9d/0x4e0
[ 52.978004] ? account_entity_enqueue+0x9c/0xd0
[ 52.978055] ___slab_alloc+0x507/0x6a0
[ 52.978097] ? create_io_worker.isra.0+0x35/0x180
[ 52.978147] ? activate_task+0x7a/0x160
[ 52.978187] ? check_preempt_curr+0x4a/0x90
[ 52.978230] ? ttwu_do_wakeup+0x19/0x140
[ 52.978273] __slab_alloc+0x1c/0x30
[ 52.978312] kmem_cache_alloc_node_trace+0xa6/0x260
[ 52.978364] create_io_worker.isra.0+0x35/0x180
[ 52.978412] io_wq_manager+0xa4/0x250
[ 52.978448] kthread+0xf9/0x130
[ 52.978479] ? create_io_worker.isra.0+0x180/0x180
[ 52.978521] ? kthread_park+0x90/0x90
[ 52.978560] ret_from_fork+0x1f/0x40
[ 52.978598] Modules linked in: ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib vfat fat ib_umad rpcrdma sunrpc rdma_ucm ib_iser rdma_cm intel_rapl_msr intel_rapl_common iw_cm ib_cm isst_if_common libiscsi scsi_transport_iscsi skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass i40iw crct10dif_pclmul crc32_pclmul iTCO_wdt ghash_clmulni_intel ib_uverbs iTCO_vendor_support intel_cstate ipmi_ssif ib_core intel_uncore joydev intel_rapl_perf ioatdma mei_me pcspkr lpc_ich i2c_i801 mei switchtec ipmi_si dca ipmi_devintf ipmi_msghandler dax_pmem dax_pmem_core acpi_power_meter acpi_pad
[ 52.978655] ip_tables xfs libcrc32c rfkill nd_pmem nd_btt ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm drm_kms_helper cec drm i40e megaraid_sas crc32c_intel nvme nvme_core nfit libnvdimm wmi pkcs8_key_parser
[ 52.994250] CR2: 0000000000002088
[ 52.995607] ---[ end trace 56b95aaef917fdfe ]---
[ 53.087074] RIP: 0010:__alloc_pages_nodemask+0x132/0x340
[ 53.087807] Code: 18 01 75 04 41 80 ce 80 89 e8 48 8b 54 24 08 8b 74 24 1c c1 e8 0c 48 8b 3c 24 83 e0 01 88 44 24 20 48 85 d2 0f 85 74 01 00 00 <3b> 77 08 0f 82 6b 01 00 00 48 89 7c 24 10 89 ea 48 8b 07 b9 00 02
[ 53.089181] RSP: 0018:ffffa48ece75bc88 EFLAGS: 00010246
[ 53.089845] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000e8e8
[ 53.090502] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000002080
[ 53.091141] RBP: 0000000000012cc0 R08: ffff8b54d83a8a00 R09: 0000000000000002
[ 53.091770] R10: 0000000000000dc0 R11: ffff8b54e0800100 R12: 0000000000000000
[ 53.092391] R13: 0000000000012cc0 R14: 0000000000000001 R15: ffff8b54e11300f0
[ 53.093020] FS: 0000000000000000(0000) GS:ffff8b54e1100000(0000) knlGS:0000000000000000
[ 53.093659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 53.094298] CR2: 0000000000002088 CR3: 000000d7e660a004 CR4: 00000000007606e0
[ 53.094947] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 53.095588] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
