Hi Sean,
On 8/17/22 8:23 AM, Sean Christopherson wrote:
On Tue, Aug 16, 2022, Gavin Shan wrote:
On 8/16/22 3:02 PM, Gavin Shan wrote:
On 8/16/22 7:42 AM, Sean Christopherson wrote:
On Mon, Aug 15, 2022, kernel test robot wrote:
commit: e923b0537d28e15c9d31ce8b38f810b325816903 ("KVM: selftests: Fix target thread to be migrated in rseq_test")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
...
# selftests: kvm: rseq_test
# ==== Test Assertion Failure ====
# rseq_test.c:278: i > (NR_TASK_MIGRATIONS / 2)
# pid=49599 tid=49599 errno=4 - Interrupted system call
# 1 0x000000000040265d: main at rseq_test.c:278
# 2 0x00007fe44eed07fc: ?? ??:0
# 3 0x00000000004026d9: _start at ??:?
# Only performed 23174 KVM_RUNs, task stalled too much?
#
not ok 56 selftests: kvm: rseq_test # exit=254
...
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 5.19.0-rc6 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="gcc-11 (Debian 11.3.0-3) 11.3.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=110300
CONFIG_CLANG_VERSION=0
CONFIG_AS_IS_GNU=y
CONFIG_AS_VERSION=23800
CONFIG_LD_IS_BFD=y
CONFIG_LD_VERSION=23800
CONFIG_LLD_VERSION=0
Assuming 23800 == 2.38, this is a known issue.
https://lore.kernel.org/all/20220810104114.6838-1-gshan@xxxxxxxxxx
It's probably different story this time.
Doh, if I had bothered to actually look at the error message...
Ok :)
The assert is triggered because of the following instructions. I would
guess the reason is vcpu thread has been running on CPU where we has high
CPU load. In this case, the vcpu thread can't be run in time. More
specific, the vcpu thread can't be run in the 1 - 10us time window, which
is specified by the migration worker (thread).
TEST_ASSERT(i > (NR_TASK_MIGRATIONS / 2),
"Only performed %d KVM_RUNs, task stalled too much?\n", i);
I think we need to improve the handshake mechanism between the vcpu thread
and migration worker. In current implementation, the handshake is done through
the atomic counter. The mechanism is simple enough, but vcpu thread can miss
the aforementioned time window. Another issue is the test case much more time
than expected to finish.
There's not really an expected time to finish. The original purpose of the test
is to trigger a kernel race condition, so it's a balance between letting the test
run long enough to have some confidence that the kernel is bug free, and not running
so long that it wastes time.
Yeah, I was thinking of it. It's why I'm not 100% sure for my proposal, to have
full synchronization.
Sean, if you think it's reasonable, I can figure out something to do:
- Reuse the atomic counter for a full synchronization between these two
threads. Something like below:
#define RSEQ_TEST_STATE_RUN_VCPU 0 // vcpu_run()
#define RSEQ_TEST_STATE_MIGRATE 1 // sched_setaffinity()
#define RSEQ_TEST_STATE_CHECK 2 // Check rseq.cpu_id and get_cpu()
The atomic counter is reset to RSEQ_TEST_STATE_RUN_VCPU after RSEQ_TEST_STATE_RUN_VCPU
Again, because one of the primary goals is to ensure the kernel is race free, the
test should avoid full synchronization.
Ok.
- Reduce NR_TASK_MIGRATIONS from 100000 to num_of_online_cpus(). With this,
less time is needed to finish the test case.
I'm able to recreate the issue on my local arm64 system.
- From the source code, the iteration count is changed from 100000 to 1000
- Only CPU#0 and CPU#1 are exposed in calc_min_max_cpu, meaning other CPUs
are cleared from @possible_mask
- Run some CPU bound task on CPU#0 and CPU#1
# while true; do taskset -c 0 ./a; done
# while true; do taskset -c 1 ./a; done
- Run 'rseq_test' and hit the issue
At this point, this isn't a test bug. The test is right to complain that it didn't
provide the coverage it's supposed to provide.
If the bot failure is a one-off, my preference is to leave things as-is for now.
If the failure is an ongoing issue, then we probably need to understand why the
bot is failing.
Yeah, the system for the coverage was likely having high CPU loads, which is similar
to my (simulated) environment. I usually have my system being idle when running the
coverage test cases. I didn't hit this specific failure before.
Lets leave it as of being. We can improve if needed in future :)
Thanks,
Gavin
