Re: Lots of systemctl segfaults in Koji Rawhide

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Tue, 2020-05-19 at 17:19 -0700, Adam Williamson wrote:
> On Tue, 2020-05-19 at 16:34 -0700, Adam Williamson wrote:
> > On Tue, 2020-05-19 at 15:45 -0700, Adam Williamson wrote:
> > > still, having trouble pinning down a culprit; it's not kernel-5.7.0-
> > > 0.rc6.1.fc33 as the 20200518.n.0 compose failed, and that was run with
> > > the previous kernel build, which *succeeded* in the 20200517.n.0
> > > compose...
> > > 
> > > I guess we get to poke through everything built around the 17th and try
> > > to find a relevant change? :)
> > 
> > This should be *more or less* the list of candidate culprits, which I
> > just heath robinson'd out of Bodhi (yet again I wish we got compose
> > reports for doomed composes, it'd make this easier...). grub2 and
> > openssl are the ones that jump out at me right away...ceph is also in
> > there, and ceph is one of the things that *is* linked to libibverbs ,
> > though the change in ceph doesn't seem like a significant one...
> > 
> > aha! This looks juicy. systemctl is linked against libpcap.so.1 , part
> > of libpcap, which is in the list below, and this is the changelog for
> > it:
> > 
> > * Fri May 15 2020 Michal Ruprich <michalruprich@xxxxxxxxx> - 14:1.9.1-4
> > - Enabling rdma support in libpcap
> > 
> > the changelog date is May 15 but the pcap build actually ran on 2020-
> > 05-18, and indeed the last successful Rawhide compose (20200517.n.1)
> > had 1.9.1-3.fc33, and the first failed compose (20200518.n.0) has
> > 1.9.1-4.fc33.
> > 
> > I'm betting we have a circular dependency problem or something here...
> 
> Well, seems like I may be half right. We untagged libcap-1.9.1-4.fc33
> and I re-ran the nbdkit build. It seems that this has fixed the "error
> while loading shared libraries: libibverbs.so.1: cannot open shared
> object file: No such file or directory" errors...but not the segfaults
> :( we now just get segfaults all the way:
> 
> DEBUG util.py:602:  Running transaction
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 1: 1976807 Segmentation fault      (core dumped) systemd-machine-id-setup &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 21: 1976809 Segmentation fault      (core dumped) systemctl daemon-reexec &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 23: 1976811 Segmentation fault      (core dumped) journalctl --update-catalog &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 24: 1976813 Segmentation fault      (core dumped) systemd-tmpfiles --create &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976821 Segmentation fault      (core dumped) systemctl preset-all &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.yHBGhZ: line 50: 1976823 Segmentation fault      (core dumped) systemctl --global preset-all &> /dev/null
> DEBUG util.py:602:  /var/tmp/rpm-tmp.TL9iz0: line 6: 1976826 Segmentation fault      (core dumped) /usr/bin/systemctl --no-reload preset dbus.socket
> 
> so there might be two different problems here, possibly?

OK, so I dug into the history of the segfaults a bit. Turns out they've
been going on for a few days. They only seem to happen on builds where
the systemd package gets pulled into the buildroot. The closest delta
I've got is this:

GOOD: https://koji.fedoraproject.org/koji/buildinfo?buildID=1507622 (Fri, 15 May 2020 18:01:07 UTC)
BAD:  https://koji.fedoraproject.org/koji/buildinfo?buildID=1507650 (Fri, 15 May 2020 19:49:31 UTC)

"GOOD" is the latest build I can find where systemd was pulled in as a
build dep (so the bug "should have" happened) and the bug didn't
happen. "BAD" is the earliest build I can find where the bug *did*
happen.

The most suspicious change between the two build envs that I can see is
openssl. GOOD has openssl-1.1.1g-1.fc33.x86_64 , and BAD has
openssl-1.1.1g-2.fc33.x86_64 . I'm gonna try doing an openssl build
with the patch from -2 reverted and see where that gets us.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux