On Fri, Jul 26, 2019 at 11:10 PM Yonghong Song <yhs@xxxxxx> wrote: > > > > On 7/26/19 2:02 PM, Sedat Dilek wrote: > > On Fri, Jul 26, 2019 at 10:38 PM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote: > >> > >> Hi Yonghong Song, > >> > >> On Fri, Jul 26, 2019 at 5:45 PM Yonghong Song <yhs@xxxxxx> wrote: > >>> > >>> > >>> > >>> On 7/26/19 1:26 AM, Sedat Dilek wrote: > >>>> Hi, > >>>> > >>>> I have opened a new issue in the ClangBuiltLinux issue tracker. > >>> > >>> Glad to know clang 9 has asm goto support and now It can compile > >>> kernel again. > >>> > >> > >> Yupp. > >> > >>>> > >>>> I am seeing a problem in the area bpf/seccomp causing > >>>> systemd/journald/udevd services to fail. > >>>> > >>>> [Fri Jul 26 08:08:43 2019] systemd[453]: systemd-udevd.service: Failed > >>>> to connect stdout to the journal socket, ignoring: Connection refused > >>>> > >>>> This happens when I use the (LLVM) LLD ld.lld-9 linker but not with > >>>> BFD linker ld.bfd on Debian/buster AMD64. > >>>> In both cases I use clang-9 (prerelease). > >>> > >>> Looks like it is a lld bug. > >>> > >>> I see the stack trace has __bpf_prog_run32() which is used by > >>> kernel bpf interpreter. Could you try to enable bpf jit > >>> sysctl net.core.bpf_jit_enable = 1 > >>> If this passed, it will prove it is interpreter related. > >>> > >> > >> After... > >> > >> sysctl -w net.core.bpf_jit_enable=1 > >> > >> I can start all failed systemd services. > >> > >> systemd-journald.service > >> systemd-udevd.service > >> haveged.service > >> > >> This is in maintenance mode. > >> > >> What is next: Do set a permanent sysctl setting for net.core.bpf_jit_enable? > >> > > > > This is what I did: > > I probably won't have cycles to debug this potential lld issue. > Maybe you already did, I suggest you put enough reproducible > details in the bug you filed against lld so they can take a look. > I understand and will put the journalctl-log into the CBL issue tracker and update informations. Thanks for your help understanding the BPF correlations. Is setting 'net.core.bpf_jit_enable = 2' helpful here? Values : 0 - disable the JIT (default value) 1 - enable the JIT 2 - enable the JIT and ask the compiler to emit traces on kernel log. Which files should LLD folks look at? cd linux find ./ -name '*bpf*.o' | grep jit ./arch/x86/net/bpf_jit_comp.o Compare the objdumps? I have archived the full build-dirs of clang9+ld.bfd and clang9+ld.lld-9. Thanks for your help! - sed@ - [1] https://sysctl-explorer.net/net/core/bpf_jit_enable/ > > > > Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for > > address: ffffffffa8203370 > > Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode > > Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page > > Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD > > 2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062 > > Jul 26 22:43:06 iniza kernel: Oops: 0000 [#3] SMP PTI > > Jul 26 22:43:06 iniza kernel: CPU: 3 PID: 436 Comm: systemd-udevd > > Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto > > #7~buster+dileks1 > > Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO > > 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019 > > Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0 > > Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c > > 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c > > c3 48 83 c3 08 0f b6 > > 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48 83 f8 1e 0f 8f c8 00 > > Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0327a88 EFLAGS: 00010246 > > Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0327b30 RBX: > > ffffb3cec00d1038 RCX: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0327b10 RSI: > > 00000000000000ac RDI: ffffb3cec0327ab0 > > Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0327aa0 R08: > > ffff9b33c94c0a00 R09: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: R10: ffff9b33cfe14e00 R11: > > ffffffffa77b8210 R12: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14: > > 0000000000000000 R15: ffffb3cec0327ab0 > > Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000) > > GS:ffff9b33d2580000(0000) knlGS:0000000000000000 > > Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3: > > 000000044f3ea006 CR4: 00000000003606e0 > > Jul 26 22:43:06 iniza kernel: Call Trace: > > Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70 > > Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60 > > Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220 > > Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100 > > Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0 > > Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0 > > Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330 > > Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110 > > Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490 > > Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0 > > Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0 > > Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0 > > Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0 > > Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0 > > Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90 > > Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914 > > Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff > > ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75 > > 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41 > > 89 d4 55 48 89 f5 53 > > Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS: > > 00000246 ORIG_RAX: 000000000000002e > > Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX: > > 0000561e28ac9390 RCX: 00007f7ac3519914 > > Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI: > > 00007ffcfb66a4a0 RDI: 000000000000000d > > Jul 26 22:43:06 iniza kernel: RBP: 0000561e28acd210 R08: > > 0000561e28990140 R09: 0000000000000002 > > Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11: > > 0000000000000246 R12: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14: > > 000000000000005e R15: 00007ffcfb66a490 > > Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss > > nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc > > efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16 > > jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod > > uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10 > > raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor > > raid6_pq libcrc32c raid1 raid0 multipath linear md_mod > > crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel > > aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper > > i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper > > xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video > > button > > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 > > Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0391 ]--- > > Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0 > > Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c > > 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c > > c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48 > > 83 f8 1e 0f 8f c8 00 > > Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec0253cb8 EFLAGS: 00010246 > > Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec0253d60 RBX: > > ffffb3cec00e9038 RCX: 0000000000000002 > > Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec0253d40 RSI: > > 00000000000000ac RDI: ffffb3cec0253ce0 > > Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec0253cd0 R08: > > 0000000000000000 R09: ffffb3cec0253f58 > > Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11: > > ffffffffa77b8210 R12: 000000007fff0000 > > Jul 26 22:43:06 iniza kernel: R13: ffffb3cec0253eb8 R14: > > 0000000000000000 R15: ffffb3cec0253ce0 > > Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000) > > GS:ffff9b33d2580000(0000) knlGS:0000000000000000 > > Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3: > > 000000044f3ea006 CR4: 00000000003606e0 > > Jul 26 22:43:06 iniza kernel: BUG: unable to handle page fault for > > address: ffffffffa8203370 > > Jul 26 22:43:06 iniza kernel: #PF: supervisor read access in kernel mode > > Jul 26 22:43:06 iniza kernel: #PF: error_code(0x0000) - not-present page > > Jul 26 22:43:06 iniza kernel: PGD 2cfa0e067 P4D 2cfa0e067 PUD > > 2cfa0f063 PMD 450829063 PTE 800ffffd30bfc062 > > Jul 26 22:43:06 iniza kernel: Oops: 0000 [#4] SMP PTI > > Jul 26 22:43:06 iniza kernel: CPU: 0 PID: 437 Comm: systemd-udevd > > Tainted: G D 5.3.0-rc1-7-amd64-cbl-asmgoto > > #7~buster+dileks1 > > Jul 26 22:43:06 iniza kernel: Hardware name: LENOVO > > 20HDCTO1WW/20HDCTO1WW, BIOS N1QET83W (1.58 ) 04/18/2019 > > Jul 26 22:43:06 iniza kernel: RIP: 0010:___bpf_prog_run+0x40/0x14f0 > > Jul 26 22:43:06 iniza kernel: Code: f3 eb 24 48 83 f8 38 0f 84 a9 0c > > 00 00 48 83 f8 39 0f 85 8a 14 00 00 0f 1f 00 48 0f bf 43 02 48 8d 1c > > c3 48 83 c3 08 0f b6 33 <48> 8b 04 f5 10 2e 20 a8 48 83 f8 3b 7f 62 48 > > 83 f8 1e 0f 8f c8 00 > > Jul 26 22:43:06 iniza kernel: RSP: 0018:ffffb3cec032fa88 EFLAGS: 00010246 > > Jul 26 22:43:06 iniza kernel: RAX: ffffb3cec032fb30 RBX: > > ffffb3cec00d1038 RCX: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: RDX: ffffb3cec032fb10 RSI: > > 00000000000000ac RDI: ffffb3cec032fab0 > > Jul 26 22:43:06 iniza kernel: RBP: ffffb3cec032faa0 R08: > > ffff9b33cf34b000 R09: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: R10: ffff9b33cf3a3400 R11: > > ffffffffa77b8210 R12: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: R13: ffffb3cec00d1000 R14: > > 0000000000000000 R15: ffffb3cec032fab0 > > Jul 26 22:43:06 iniza kernel: FS: 00007f7ac2d28d40(0000) > > GS:ffff9b33d2400000(0000) knlGS:0000000000000000 > > Jul 26 22:43:06 iniza kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 CR3: > > 000000044724a001 CR4: 00000000003606f0 > > Jul 26 22:43:06 iniza kernel: Call Trace: > > Jul 26 22:43:06 iniza kernel: __bpf_prog_run32+0x44/0x70 > > Jul 26 22:43:06 iniza kernel: ? prep_new_page+0x47/0x1a0 > > Jul 26 22:43:06 iniza kernel: ? security_sock_rcv_skb+0x3f/0x60 > > Jul 26 22:43:06 iniza kernel: sk_filter_trim_cap+0xe4/0x220 > > Jul 26 22:43:06 iniza kernel: ? __skb_clone+0x2e/0x100 > > Jul 26 22:43:06 iniza kernel: netlink_broadcast_filtered+0x2df/0x4f0 > > Jul 26 22:43:06 iniza kernel: netlink_sendmsg+0x34f/0x3c0 > > Jul 26 22:43:06 iniza kernel: ___sys_sendmsg+0x315/0x330 > > Jul 26 22:43:06 iniza kernel: ? seccomp_run_filters+0x54/0x110 > > Jul 26 22:43:06 iniza kernel: ? filename_parentat+0x210/0x490 > > Jul 26 22:43:06 iniza kernel: ? __seccomp_filter+0xf7/0x6e0 > > Jul 26 22:43:06 iniza kernel: ? __d_alloc+0x159/0x1c0 > > Jul 26 22:43:06 iniza kernel: ? kmem_cache_free+0x1e/0x5c0 > > Jul 26 22:43:06 iniza kernel: ? fast_dput+0x73/0xb0 > > Jul 26 22:43:06 iniza kernel: __x64_sys_sendmsg+0x97/0xe0 > > Jul 26 22:43:06 iniza kernel: do_syscall_64+0x59/0x90 > > Jul 26 22:43:06 iniza kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 > > Jul 26 22:43:06 iniza kernel: RIP: 0033:0x7f7ac3519914 > > Jul 26 22:43:06 iniza kernel: Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff > > ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 e9 5d 0c 00 8b 00 85 c0 75 > > 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 41 > > 89 d4 55 48 89 f5 53 > > Jul 26 22:43:06 iniza kernel: RSP: 002b:00007ffcfb66a478 EFLAGS: > > 00000246 ORIG_RAX: 000000000000002e > > Jul 26 22:43:06 iniza kernel: RAX: ffffffffffffffda RBX: > > 0000561e28aaa600 RCX: 00007f7ac3519914 > > Jul 26 22:43:06 iniza kernel: RDX: 0000000000000000 RSI: > > 00007ffcfb66a4a0 RDI: 000000000000000e > > Jul 26 22:43:06 iniza kernel: RBP: 0000561e28aaaac0 R08: > > 0000561e28990140 R09: 0000000000000002 > > Jul 26 22:43:06 iniza kernel: R10: 0000000000000000 R11: > > 0000000000000246 R12: 0000000000000000 > > Jul 26 22:43:06 iniza kernel: R13: 0000000000000000 R14: > > 000000000000005d R15: 00007ffcfb66a490 > > Jul 26 22:43:06 iniza kernel: Modules linked in: nfsd auth_rpcgss > > nfs_acl lockd grace i2c_dev parport_pc ppdev lp parport sunrpc > > efivarfs ip_tables x_tables autofs4 ext4 crc32c_generic mbcache crc16 > > jbd2 btrfs zstd_decompress zstd_compress algif_skcipher af_alg sd_mod > > uas usb_storage scsi_mod hid_generic usbhid hid dm_crypt dm_mod raid10 > > raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor > > raid6_pq libcrc32c raid1 raid0 multipath linear md_mod > > crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel > > aesni_intel i915 intel_lpss_pci nvme aes_x86_64 glue_helper > > i2c_algo_bit crypto_simd cryptd xhci_pci psmouse e1000e drm_kms_helper > > xhci_hcd i2c_i801 nvme_core intel_lpss drm usbcore thermal wmi video > > button > > Jul 26 22:43:06 iniza kernel: CR2: ffffffffa8203370 > > Jul 26 22:43:06 iniza kernel: ---[ end trace 312670b063bd0392 ]--- > > > > Full `journalctl -xb` attached. > > > > - Sedat - > >
