Hello,
On 4/14/2023 10:41 AM, Jain, Ayush wrote:
Hello Broonie,
When Running fio-test on latest linux-next tree, I noticed that test hung indefinitely, Going back I see that this problem exists since
next-20230316 release, After bisecting I landed on the following merge commit by Jens.
Commit 097d3ca138f9 ("Merge branch 'for-6.4/splice' into for-next")
Running perf I see following trace and call-stack for fio:
Overhead Command Shared Object Symbol
25.08% fio [kernel.vmlinux] [k] copy_user_generic_string
copy_user_generic_string
__do_splice
__x64_sys_splice
do_syscall_64
entry_SYSCALL_64_after_hwframe
splice
0x1c44be0
...
On a good kernel I see the following perf trace:
Overhead Command Shared Object Symbol
49.93% fio fio [.] fio_crc32
7.23% fio fio [.] clock_thread_fn
2.10% fio [kernel.vmlinux] [k] clear_page_rep
1.55% fio fio [.] __fill_random_buf
1.35% fio [kernel.vmlinux] [k] loop_queue_rq
1.05% fio [kernel.vmlinux] [k] copy_user_generic_string
...
I see some splice changes being added as the part of merge
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=for-6.4/splice
I observe this problem on 3 EPYC system(Zen1,3,4), with the following disk architecture
Zen1: nvme0n1 931.5G Samsung SSD 970 EVO Plus 1TB
Zen4: nvme0n1 232.9G Samsung SSD 960 EVO 250GB
I am running fio as follows:
$fio fio-simple.job --filename=/dev/test_vg/test_lv
where test_lv is mounted as follows:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 12.2G 0 loop
└─md127 9:127 0 12.2G 0 raid0
└─test_vg-test_lv 253:3 0 5.5G 0 lvm
You can find fio-simple.job at
https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/io/disk/fiotest.py.data/fio-simple.job
Fio Version: fio-3.34-25-g07ed
Also adding to these observations
-If we create a filesystem on the raw disk -- Test completes with a Pass
-If there is no Filesystem on the raw disk(loop, nvme) -- Test hangs with the provided trace
Regards
Ayush Jain
> On 4/13/2023 11:55 PM, broonie@xxxxxxxxxx wrote:
Hi all,
Changes since 20230411:
The sh tree gained a conflict with the mm-unstable tree.
The erofs tree gained a conflict with the vfs-idmapping tree.
The ext4 tree gained multiple conflicts with the mm-stable tree.
The net-next tree gained a conflict with the origin tree.
The bpf-next tree gained a conflict with the net-net tree.
Non-merge commits (relative to Linus' tree): 10382
11028 files changed, 580165 insertions(+), 249563 deletions(-)
----------------------------------------------------------------------------
I have created today's linux-next tree at
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
(patches at http://www.kernel.org/pub/linux/kernel/next/ ). If you
are tracking the linux-next tree using git, you should not use "git pull"
to do so as that will try to merge the new linux-next release with the
old one. You should use "git fetch" and checkout or reset to the new
master.
You can see which trees have been included by looking in the Next/Trees
file in the source. There is also the merge.log file in the Next
directory. Between each merge, the tree was built with a ppc64_defconfig
for powerpc, an allmodconfig for x86_64, a multi_v7_defconfig for arm
and a native build of tools/perf. After the final fixups (if any), I do
an x86_64 modules_install followed by builds for x86_64 allnoconfig,
powerpc allnoconfig (32 and 64 bit), ppc44x_defconfig, allyesconfig
and pseries_le_defconfig and i386, arm64, s390, sparc and sparc64
defconfig and htmldocs. And finally, a simple boot test of the powerpc
pseries_le_defconfig kernel in qemu (with and without kvm enabled).
Below is a summary of the state of the merge.
I am currently merging 357 trees (counting Linus' and 102 trees of bug
fix patches pending for the current merge release).
Stats about the size of the tree over time can be seen at
http://neuling.org/linux-next-size.html .
Status of my local build tests will be at
http://kisskb.ellerman.id.au/linux-next . If maintainers want to give
advice about cross compilers/configs that work, we are always open to add
more builds.
Thanks to Randy Dunlap for doing many randconfig builds. And to Paul
Gortmaker for triage and bug fixes.
Regards,
Ayush Jain
