> FWIW, this is now upstream and will land in 6.7, together with a similar
> patch which adds user stacktrace support:
> 504b73d00a55 ("s390/perf: implement perf_callchain_user()")
> aa44433ac4ee ("s390: add USER_STACKTRACE support")
> Please let us know if there are any problems.
I've been trying out the new userspace stacktrace support on Fedora
Rawhide but I'm having some problems getting it to work. I rebuilt
systemd and all of its dependencies (specifically, glibc) with
-mbackchain using copr
(https://copr.fedorainfracloud.org/coprs/daandemeyer/fno-omit-frame-pointer/builds/),
built a Fedora Rawhide image with the rebuilt packages and perf using
mkosi, started an s390x virtual machine and ran "perf record -g
journalctl --verify" followed by "perf report -g 'graph,0.5,caller'".
On x86, this gives me a sensible perf report. On s390x, it seems a lot
of information is still missing.
uname -r: 6.7.0-0.rc6.48.fc40.s390x
cat /usr/lib/modules/6.7.0-0.rc6.48.fc40.s390x/config | grep STACK:
"""
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_STACKLEAK=y
CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK=y
CONFIG_SOFTIRQ_ON_OWN_STACK=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=y
CONFIG_RANDOMIZE_KSTACK_OFFSET=y
CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=y
CONFIG_BLK_MQ_STACKING=y
CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y
CONFIG_NET_EMATCH_STACK=32
# CONFIG_INIT_STACK_NONE is not set
# CONFIG_INIT_STACK_ALL_PATTERN is not set
CONFIG_INIT_STACK_ALL_ZERO=y
CONFIG_ARCH_STACKWALK=y
CONFIG_STACKDEPOT=y
# CONFIG_STACKTRACE_BUILD_ID is not set
# CONFIG_DEBUG_STACK_USAGE is not set
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_STACKTRACE=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_STACK_TRACER=y
CONFIG_STACKINIT_KUNIT_TEST=m
"""
To reproduce (from a s390x host system):
- git clone https://www.github.com/systemd/mkosi
- cd mkosi
- tee copr.repo <<EOF
[copr:copr.fedorainfracloud.org:daandemeyer:fno-omit-frame-pointer]
name=Copr repo for fno-omit-frame-pointer owned by daandemeyer
baseurl=https://download.copr.fedorainfracloud.org/results/daandemeyer/fno-omit-frame-pointer/fedora-$releasever-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/daandemeyer/fno-omit-frame-pointer/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1
priority=1
EOF
- tee mkosi.local.conf <<EOF
[Distribution]
Distribution=fedora
Release=rawhide
Repositories=fedora-debuginfo
PackageManagerTrees=copr.repo:/etc/yum.repos.d/copr.repo
[Content]
Packages=
systemd-debuginfo
systemd-debugsource
[Host]
ToolsTree=default
Incremental=yes
EOF
- bin/mkosi -f qemu"
# You'll end up in a virtual machine running Fedora Rawhide with a
6.7-rc kernel and glibc + systemd built with -mbackchain
- perf record -g journalctl --verify
- perf report -g 'graph,0.5,caller'
# The output will not make a lot of sense whereas it does on x86
On Thu, 9 Nov 2023 at 15:48, Heiko Carstens <hca@xxxxxxxxxxxxx> wrote:
>
> On Mon, Oct 30, 2023 at 09:19:02AM -0400, Neal Gompa wrote:
> > On Mon, Oct 30, 2023 at 9:02 AM Heiko Carstens <hca@xxxxxxxxxxxxx> wrote:
> > >
> > > On Fri, Oct 27, 2023 at 11:22:42AM -0400, Neal Gompa wrote:
> > > > On Fri, Oct 27, 2023 at 6:10 AM Heiko Carstens <hca@xxxxxxxxxxxxx> wrote:
> > > > >
> > > > > On Fri, Oct 27, 2023 at 10:00:53AM +0200, Daan De Meyer wrote:
> > > > > >
> > > > > > If the kernel gets support for s390x user space unwinding using the backchain,
> > > > > > we'll propose to enable -mbackchain in the default compilation flags for Fedora
> > > > > > so that s390x on Fedora will have the same profiling experience as x86-64, arm64
> > > > > > and ppc64. For now we'll keep the status quo since compiling with the backchain
> > > > > > doesn't provide any benefit until the kernel unwinder can unwind user
> > > > > > space stacks
> > > > > > using it.
> > > > > >
> > > > > > Thanks for clarifying the current state of user space stack unwinding on s390x!
> > > > >
> > > > > We will implement the missing pieces and let you know when things are
> > > > > supposed to work.
> > > >
> > > > Do you think we could have an initial patch set for implementing the
> > > > missing pieces in time for the Linux 6.8 merge window? Then we can
> > > > look at enabling this for s390x as a Fedora Linux 40 Change.
> > >
> > > This will be very likely the case. Actually the plan is to go with the
> > > patch below. I gave it some testing with Fedora 38 and replaced (only)
> > > glibc with a variant that was compiled with -mbackchain.
> ...
> > This patch LGTM. I'd love to see it land in Linux 6.7!
> >
> > Reviewed-by: Neal Gompa <ngompa@xxxxxxxxxxxxxxxxx>
>
> FWIW, this is now upstream and will land in 6.7, together with a similar
> patch which adds user stacktrace support:
>
> 504b73d00a55 ("s390/perf: implement perf_callchain_user()")
> aa44433ac4ee ("s390: add USER_STACKTRACE support")
>
> Please let us know if there are any problems.
