On 2019/07/25 19:53, Brian Foster wrote:
> This is a known problem. XFS delayed allocation has a window between
> delalloc to real block conversion and writeback completion where stale
> data exposure is possible if the writeback doesn't complete (i.e., due
> to crash, I/O error, etc.). See fstests generic/536 for another
> reference. We've batted around potential solutions like using unwritten
> extents for delalloc allocations, but IIRC we haven't been able to come
> up with something with suitable performance to this point.
>
> I'm curious why your OOM test results in writeback errors in the first
> place. Is that generally expected? Does dmesg show any other XFS related
> events, such as filesystem shutdown for example? I gave it a quick try
> on a 4GB swapless VM and it doesn't trigger OOM. What's your memory
> configuration and what does the /tmp filesystem look like ('xfs_info
> /tmp')?
Writeback errors should not happen by just close-to-OOM situation.
And there is no other XFS related events.
----------
[ 828.600750][ T5241] oom-torture invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[ 828.608041][ T5241] CPU: 1 PID: 5241 Comm: oom-torture Not tainted 5.3.0-rc1+ #626
[ 828.612160][ T5241] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
[ 828.617700][ T5241] Call Trace:
[ 828.619616][ T5241] dump_stack+0x67/0x95
[ 828.621791][ T5241] dump_header+0x4d/0x3e0
[ 828.624026][ T5241] oom_kill_process+0x193/0x220
[ 828.626412][ T5241] out_of_memory+0x105/0x360
[ 828.628797][ T5241] ? out_of_memory+0x1ab/0x360
[ 828.631133][ T5241] __alloc_pages_slowpath+0x937/0xbce
[ 828.633676][ T5241] __alloc_pages_nodemask+0x372/0x3b0
[ 828.636181][ T5241] do_anonymous_page+0xe0/0x5e0
[ 828.638542][ T5241] __handle_mm_fault+0x8d6/0xbe0
[ 828.640926][ T5241] handle_mm_fault+0x179/0x380
[ 828.643228][ T5241] ? handle_mm_fault+0x46/0x380
[ 828.645546][ T5241] __do_page_fault+0x255/0x4d0
[ 828.647753][ T5241] do_page_fault+0x27/0x1e0
[ 828.649859][ T5241] page_fault+0x34/0x40
[ 828.651820][ T5241] RIP: 0033:0x40082f
[ 828.653687][ T5241] Code: 31 e4 e8 54 ff ff ff 31 c0 48 85 db 75 15 eb 3c 0f 1f 00 49 8d 84 24 00 10 00 00 48 39 c3 76 2c 49 89 c4 80 3d 91 18 20 00 00 <c6> 44 05 00 00 74 e2 31 f6 31 ff ba 0a 00 00 00 e8 ec fe ff ff 49
[ 828.661900][ T5241] RSP: 002b:00007ffc18ab7fe0 EFLAGS: 00010202
[ 828.664423][ T5241] RAX: 00000000c43f9000 RBX: 0000000100000000 RCX: 00007ff8e4083370
[ 828.667648][ T5241] RDX: 000000000000000a RSI: 0000000000000000 RDI: 0000000000000000
[ 828.670833][ T5241] RBP: 00007ff6e4184010 R08: 00007ffc18ab7f10 R09: 00007ffc18ab7d50
[ 828.674029][ T5241] R10: 00007ffc18ab7a60 R11: 0000000000000246 R12: 00000000c43f9000
[ 828.677236][ T5241] R13: 00007ff6e4184010 R14: 0000000000000000 R15: 0000000000000000
[ 828.685651][ T5241] Mem-Info:
[ 828.690368][ T5241] active_anon:808378 inactive_anon:3716 isolated_anon:0
[ 828.690368][ T5241] active_file:434 inactive_file:583 isolated_file:0
[ 828.690368][ T5241] unevictable:0 dirty:85 writeback:108 unstable:0
[ 828.690368][ T5241] slab_reclaimable:8616 slab_unreclaimable:28454
[ 828.690368][ T5241] mapped:2428 shmem:4221 pagetables:5031 bounce:0
[ 828.690368][ T5241] free:25443 free_pcp:646 free_cma:0
[ 828.713486][ T5241] Node 0 active_anon:3233512kB inactive_anon:14864kB active_file:1784kB inactive_file:2744kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:9712kB dirty:324kB writeback:532kB shmem:16884kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2826240kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 828.730876][ T5241] DMA free:14272kB min:296kB low:368kB high:440kB active_anon:1632kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 828.743754][ T5241] lowmem_reserve[]: 0 2679 3495 3495
[ 828.752288][ T5241] DMA32 free:54488kB min:51572kB low:64464kB high:77356kB active_anon:2682892kB inactive_anon:0kB active_file:0kB inactive_file:544kB unevictable:0kB writepending:0kB present:3129216kB managed:2743640kB mlocked:0kB kernel_stack:2064kB pagetables:28kB bounce:0kB free_pcp:96kB local_pcp:0kB free_cma:0kB
[ 828.765008][ T5241] lowmem_reserve[]: 0 0 816 816
[ 828.770282][ T5241] Normal free:33040kB min:15712kB low:19640kB high:23568kB active_anon:548988kB inactive_anon:14864kB active_file:1608kB inactive_file:2660kB unevictable:0kB writepending:76kB present:1048576kB managed:835884kB mlocked:0kB kernel_stack:17396kB pagetables:20096kB bounce:0kB free_pcp:2348kB local_pcp:152kB free_cma:0kB
[ 828.783430][ T5241] lowmem_reserve[]: 0 0 0 0
[ 828.791481][ T5241] DMA: 2*4kB (M) 1*8kB (M) 1*16kB (M) 1*32kB (U) 2*64kB (U) 2*128kB (UM) 2*256kB (UM) 0*512kB 1*1024kB (U) 0*2048kB 3*4096kB (M) = 14272kB
[ 828.802502][ T5241] DMA32: 4*4kB (UM) 7*8kB (UM) 34*16kB (UM) 90*32kB (UME) 54*64kB (UME) 30*128kB (UME) 10*256kB (UE) 26*512kB (UM) 27*1024kB (ME) 0*2048kB 0*4096kB = 54312kB
[ 828.810562][ T5241] Normal: 856*4kB (UM) 750*8kB (UME) 589*16kB (UME) 348*32kB (UM) 3*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 30176kB
[ 828.818691][ T5241] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 828.825729][ T5241] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 828.837586][ T5241] 5941 total pagecache pages
[ 828.842234][ T5241] 0 pages in swap cache
[ 828.845175][ T5241] Swap cache stats: add 0, delete 0, find 0/0
[ 828.848688][ T5241] Free swap = 0kB
[ 828.851230][ T5241] Total swap = 0kB
[ 828.853768][ T5241] 1048445 pages RAM
[ 828.856735][ T5241] 0 pages HighMem/MovableOnly
[ 828.859683][ T5241] 149588 pages reserved
[ 828.862394][ T5241] 0 pages cma reserved
[ 828.865161][ T5241] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=oom-torture,pid=5701,uid=0
[ 828.871540][ T5241] Out of memory: Killed process 5701 (oom-torture) total-vm:4220kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB
[ 828.877783][ T35] oom_reaper: reaped process 5701 (oom-torture), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 829.475335][ T5301] XFS (sda1): writeback error on sector 90663248
[ 829.475618][ T5622] XFS (sda1): writeback error on sector 92712224
[ 829.476211][ T5557] XFS (sda1): writeback error on sector 370398432
[ 829.476801][ T5309] XFS (sda1): writeback error on sector 494252360
[ 829.477005][ T5468] XFS (sda1): writeback error on sector 493980928
[ 829.727478][ T5241] oom-torture invoked oom-killer: gfp_mask=0x100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
[ 829.734776][ T5241] CPU: 2 PID: 5241 Comm: oom-torture Not tainted 5.3.0-rc1+ #626
----------
Kernel config is http://I-love.SAKURA.ne.jp/tmp/config-5.3-rc1 .
Below result is from a different VM which shows the same problem.
# xfs_info /tmp
meta-data=/dev/sda1 isize=256 agcount=4, agsize=16383936 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0 spinodes=0
data = bsize=4096 blocks=65535744, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=31999, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
