Greetings XFS community,
We had an RCU stall [1]. According to the code, it happened in
radix_tree_gang_lookup_tag():
rcu_read_lock();
nr_found = radix_tree_gang_lookup_tag(
&pag->pag_ici_root,
(void **)batch, first_index,
XFS_LOOKUP_BATCH,
XFS_ICI_RECLAIM_TAG);
This XFS system has over 100M files. So perhaps looping inside the radix
tree took too long, and it was happening in RCU read-side critical seciton.
This is one of the possible causes for RCU stall.
This happened in kernel 4.14.99, but looking at latest mainline code, code
is still the same.
Can anyone please advise how to address that? It is not possible to put
cond_resched() inside the radix tree code, because it can be used with
spinlocks, and perhaps other contexts where sleeping is not allowed.
Thanks,
Alex.
[1]
Nov 16 12:33:15.314001 vsa-0000002a-vc-1 kernel: [78961.809221] INFO:
rcu_preempt detected stalls on CPUs/tasks:
Nov 16 12:33:15.314031 vsa-0000002a-vc-1 kernel: [78961.810785] Tasks
blocked on level-0 rcu_node (CPUs 0-4): P8854
Nov 16 12:33:15.314033 vsa-0000002a-vc-1 kernel: [78961.812141]
(detected by 3, t=60004 jiffies, g=18947673, c=18947672, q=110854)
Nov 16 12:33:15.314036 vsa-0000002a-vc-1 kernel: [78961.814050] python
R running task 0 8854 8792 0x00000000
Nov 16 12:33:15.314038 vsa-0000002a-vc-1 kernel: [78961.814054] Call Trace:
Nov 16 12:33:15.314040 vsa-0000002a-vc-1 kernel: [78961.814067] ?
__schedule+0x290/0x8a0
Nov 16 12:33:15.314041 vsa-0000002a-vc-1 kernel: [78961.814070]
preempt_schedule_irq+0x37/0x60
Nov 16 12:33:15.314043 vsa-0000002a-vc-1 kernel: [78961.814074]
retint_kernel+0x1b/0x1d
Nov 16 12:33:15.314044 vsa-0000002a-vc-1 kernel: [78961.814079] RIP:
0010:radix_tree_next_chunk+0x111/0x320
Nov 16 12:33:15.314045 vsa-0000002a-vc-1 kernel: [78961.814080] RSP:
0018:ffffb31f5fbab7d0 EFLAGS: 00000287 ORIG_RAX: ffffffffffffff10
Nov 16 12:33:15.314047 vsa-0000002a-vc-1 kernel: [78961.814082] RAX:
0000000000000033 RBX: 0000000000000010 RCX: 000000000000000c
Nov 16 12:33:15.314049 vsa-0000002a-vc-1 kernel: [78961.814083] RDX:
0000000000000000 RSI: ffffb31f5fbab818 RDI: 00000000000b3000
Nov 16 12:33:15.314051 vsa-0000002a-vc-1 kernel: [78961.814084] RBP:
ffffa07dd6d80b48 R08: ffffa08079fe56d0 R09: ffffa08079fe6481
Nov 16 12:33:15.314052 vsa-0000002a-vc-1 kernel: [78961.814085] R10:
000000000000000c R11: ffffa08079fe5890 R12: 000000000000000c
Nov 16 12:33:15.314054 vsa-0000002a-vc-1 kernel: [78961.814086] R13:
0000000000000040 R14: 0000000000000228 R15: 0000000000033000
Nov 16 12:33:15.314055 vsa-0000002a-vc-1 kernel: [78961.814090]
radix_tree_gang_lookup_tag+0xd6/0x160
Nov 16 12:33:15.314056 vsa-0000002a-vc-1 kernel: [78961.814152]
xfs_reclaim_inodes_ag+0xd5/0x310 [xfs]
Nov 16 12:33:15.314058 vsa-0000002a-vc-1 kernel: [78961.814199]
xfs_reclaim_inodes_nr+0x3c/0x50 [xfs]
Nov 16 12:33:15.314059 vsa-0000002a-vc-1 kernel: [78961.814204]
super_cache_scan+0x156/0x1a0
Nov 16 12:33:15.314061 vsa-0000002a-vc-1 kernel: [78961.814210]
shrink_slab.part.51+0x1e8/0x400
Nov 16 12:33:15.314062 vsa-0000002a-vc-1 kernel: [78961.814213]
shrink_node+0x123/0x310
Nov 16 12:33:15.314063 vsa-0000002a-vc-1 kernel: [78961.814216]
do_try_to_free_pages+0xc3/0x330
Nov 16 12:33:15.314064 vsa-0000002a-vc-1 kernel: [78961.814218]
try_to_free_pages+0xf4/0x1f0
Nov 16 12:33:15.314065 vsa-0000002a-vc-1 kernel: [78961.814224]
__alloc_pages_slowpath+0x3a5/0xd50
Nov 16 12:33:15.314067 vsa-0000002a-vc-1 kernel: [78961.814228]
__alloc_pages_nodemask+0x2a7/0x2d0
Nov 16 12:33:15.314068 vsa-0000002a-vc-1 kernel: [78961.814232]
alloc_pages_vma+0x7c/0x1e0
Nov 16 12:33:15.314070 vsa-0000002a-vc-1 kernel: [78961.814236]
__handle_mm_fault+0x90c/0x12f0
Nov 16 12:33:15.314071 vsa-0000002a-vc-1 kernel: [78961.814239]
handle_mm_fault+0xb1/0x1e0
Nov 16 12:33:15.314072 vsa-0000002a-vc-1 kernel: [78961.814244]
__do_page_fault+0x278/0x530
Nov 16 12:33:15.314073 vsa-0000002a-vc-1 kernel: [78961.814247] ?
async_page_fault+0x2f/0x50
Nov 16 12:33:15.314074 vsa-0000002a-vc-1 kernel: [78961.814248]
async_page_fault+0x45/0x50
