On 2023/12/1 01:39, James Morse wrote:
> Hi Shuai,
>
> On 07/10/2023 08:28, Shuai Xue wrote:
>> Hardware errors could be signaled by synchronous interrupt,
>
> I'm struggling with 'synchronous interrupt'. Do you mean arm64's 'precise' (all
> instructions before the exception were executed, and none after).
> Otherwise, surely any interrupt from a background scrubber is inherently asynchronous!
>
I am sorry, this is typo. I mean asynchronous interrupt.
>
>> e.g. when an
>> error is detected by a background scrubber, or signaled by synchronous
>> exception, e.g. when an uncorrected error is consumed. Both synchronous and
>> asynchronous error are queued and handled by a dedicated kthread in
>> workqueue.
>>
>> commit 7f17b4a121d0 ("ACPI: APEI: Kick the memory_failure() queue for
>> synchronous errors") keep track of whether memory_failure() work was
>> queued, and make task_work pending to flush out the workqueue so that the
>> work for synchronous error is processed before returning to user-space.
>
> It does it regardless, if user-space was interrupted by APEI any work queued as a result
> of that should be completed before we go back to user-space. Otherwise we can bounce
> between user-space and firmware, with the kernel only running the APEI code, and never
> making progress.
>
Agreed.
>
>> The trick ensures that the corrupted page is unmapped and poisoned. And
>> after returning to user-space, the task starts at current instruction which
>> triggering a page fault in which kernel will send SIGBUS to current process
>> due to VM_FAULT_HWPOISON.
>>
>> However, the memory failure recovery for hwpoison-aware mechanisms does not
>> work as expected. For example, hwpoison-aware user-space processes like
>> QEMU register their customized SIGBUS handler and enable early kill mode by
>> seting PF_MCE_EARLY at initialization. Then the kernel will directly notify
>
> (setting, directly)
Thank you. Will fix it.
>
>> the process by sending a SIGBUS signal in memory failure with wrong
>
>> si_code: the actual user-space process accessing the corrupt memory
>> location, but its memory failure work is handled in a kthread context, so
>> it will send SIGBUS with BUS_MCEERR_AO si_code to the actual user-space
>> process instead of BUS_MCEERR_AR in kill_proc().
>
> This is hard to parse, "the user-space process is accessing"? (dropping 'actual' and
> adding 'is')
Will fix it.
>
>
> Wasn't this behaviour fixed by the previous patch?
>
> What problem are you fixing here?
Nope. The memory_failure() runs in a kthread context, but not the
user-space process which consuming poison data.
// kill_proc() in memory-failure.c
if ((flags & MF_ACTION_REQUIRED) && (t == current))
ret = force_sig_mceerr(BUS_MCEERR_AR,
(void __user *)tk->addr, addr_lsb);
else
ret = send_sig_mceerr(BUS_MCEERR_AO, (void __user *)tk->addr,
addr_lsb, t);
So, even we queue memory_failure() with MF_ACTION_REQUIRED flags in
previous patch, it will still send a sigbus with BUS_MCEERR_AO in the else
branch of kill_proc().
>
>
>> To this end, separate synchronous and asynchronous error handling into
>> different paths like X86 platform does:
>>
>> - valid synchronous errors: queue a task_work to synchronously send SIGBUS
>> before ret_to_user.
>
>> - valid asynchronous errors: queue a work into workqueue to asynchronously
>> handle memory failure.
>
> Why? The signal issue was fixed by the previous patch. Why delay the handling of a
> poisoned memory location further?
The signal issue is not fixed completely. See my reply above.
>
>
>> - abnormal branches such as invalid PA, unexpected severity, no memory
>> failure config support, invalid GUID section, OOM, etc.
>
> ... do what?
If no memory failure work is queued for abnormal errors, do a force kill.
Will also add this comment to commit log.
>
>
>> Then for valid synchronous errors, the current context in memory failure is
>> exactly belongs to the task consuming poison data and it will send SIBBUS
>> with proper si_code.
>
>
>> diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
>> index 6f35f724cc14..1675ff77033d 100644
>> --- a/arch/x86/kernel/cpu/mce/core.c
>> +++ b/arch/x86/kernel/cpu/mce/core.c
>> @@ -1334,17 +1334,10 @@ static void kill_me_maybe(struct callback_head *cb)
>> return;
>> }
>>
>> - /*
>> - * -EHWPOISON from memory_failure() means that it already sent SIGBUS
>> - * to the current process with the proper error info,
>> - * -EOPNOTSUPP means hwpoison_filter() filtered the error event,
>> - *
>> - * In both cases, no further processing is required.
>> - */
>> if (ret == -EHWPOISON || ret == -EOPNOTSUPP)
>> return;
>>
>> - pr_err("Memory error not recovered");
>> + pr_err("Sending SIGBUS to current task due to memory error not recovered");
>> kill_me_now(cb);
>> }
>>
>
> I'm not sure how this hunk is relevant to the commit message.
I handle memory_failure() error code in its arm64 call site
memory_failure_cb() with some comments, similar to x86 call site
kill_me_maybe(). I moved these two part comments to function declaration,
followed by review comments from Kefeng.
I should split this into a separate patch. Will do it in next version.
>
>
>> diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
>> index 88178aa6222d..014401a65ed5 100644
>> --- a/drivers/acpi/apei/ghes.c
>> +++ b/drivers/acpi/apei/ghes.c
>> @@ -484,6 +497,18 @@ static bool ghes_do_memory_failure(u64 physical_addr, int flags)
>> return false;
>> }
>>
>> + if (flags == MF_ACTION_REQUIRED && current->mm) {
>> + twcb = kmalloc(sizeof(*twcb), GFP_ATOMIC);
>> + if (!twcb)
>> + return false;
>
> Yuck - New failure modes! This is why the existing code always has this memory allocated
> in struct ghes_estatus_node.
Are you suggesting to move fields of struct sync_task_work to struct
ghes_estatus_node, and use ghes_estatus_node here? Or we can just alloc
struct sync_task_work with gen_pool_alloc from ghes_estatus_pool.
>
>
>> + twcb->pfn = pfn;
>> + twcb->flags = flags;
>> + init_task_work(&twcb->twork, memory_failure_cb);
>> + task_work_add(current, &twcb->twork, TWA_RESUME);
>> + return true;
>> + }
>> +
>> memory_failure_queue(pfn, flags);
>> return true;
>> }
>
> [..]
>
>> @@ -696,7 +721,14 @@ static bool ghes_do_proc(struct ghes *ghes,
>> }
>> }
>>
>> - return queued;
>> + /*
>> + * If no memory failure work is queued for abnormal synchronous
>> + * errors, do a force kill.
>> + */
>> + if (sync && !queued) {
>> + pr_err("Sending SIGBUS to current task due to memory error not recovered");
>> + force_sig(SIGBUS);
>> + }
>> }
>
> I think this is a lot of churn, and this hunk is the the only meaningful change in
> behaviour. Can you explain how this happens?
For example:
- invalid GUID section in ghes_do_proc()
- CPER_MEM_VALID_PA is not set, unexpected severity in
ghes_handle_memory_failure().
- CONFIG_ACPI_APEI_MEMORY_FAILURE is not enabled, !pfn_vaild(pfn) in
ghes_do_memory_failure()
>
>
> Wouldn't it be simpler to split ghes_kick_task_work() to have a sync/async version.
> The synchronous version can unconditionally force_sig_mceerr(BUS_MCEERR_AR, ...) after
> memory_failure_queue_kick() - but that still means memory_failure() is unable to disappear
> errors that it fixed - see MF_RECOVERED.
Sorry, I don't think so. Unconditionally send a sigbus is not a good
choice. For example, if a sync memory error detected in instruction memory
error, the kernel should transparently fix and no signal should be send.
./einj_mem_uc instr
[168522.751671] Memory failure: 0x89dedd: corrupted page was clean: dropped without side effects
[168522.751679] Memory failure: 0x89dedd: recovery action for clean LRU page: Recovered
With this patch set, the instr case behaves consistently on both the arm64 and x86 platforms.
The complex page error_states are handled in memory_failure(). IMHO, we
should left this part to it.
>
>> diff --git a/mm/memory-failure.c b/mm/memory-failure.c
>> index 4d6e43c88489..0d02f8a0b556 100644
>> --- a/mm/memory-failure.c
>> +++ b/mm/memory-failure.c
>> @@ -2161,9 +2161,12 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags,
>> * Must run in process context (e.g. a work queue) with interrupts
>> * enabled and no spinlocks held.
>> *
>> - * Return: 0 for successfully handled the memory error,
>> - * -EOPNOTSUPP for hwpoison_filter() filtered the error event,
>> - * < 0(except -EOPNOTSUPP) on failure.
>> + * Return values:
>> + * 0 - success
>> + * -EOPNOTSUPP - hwpoison_filter() filtered the error event.
>> + * -EHWPOISON - sent SIGBUS to the current process with the proper
>> + * error info by kill_accessing_process().
>> + * other negative values - failure
>> */
>> int memory_failure(unsigned long pfn, int flags)
>> {
>
> I'm not sure how this hunk is relevant to the commit message.
As mentioned, I will split this into a separate patch.
>
>
> Thanks,
>
> James
Thank you for valuable comments.
Best Regards,
Shuai
