Hi, I run a PC Engines APU in my basement for local infrastructure with KVM. Kernels 4.2.4 and 4.2.5 panic on bootup pretty fast after the usual "x86: Booted up 1 node, 2 CPUs", "smpboot: Total of 2 processors activated (3999.89 BogoMIPS)" message. The message I would expect instead of the kernel panic would be "devtmpfs: initialized", as one can see on a successful 4.2.3 boot with identical kernel configuration. Loading Linux 4.2.5-zgsrv20080 ... Loading initial ramdisk ... Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Initializing cgroup subsys cpuacct Linux version 4.2.5-zgsrv20080 (mh@fan) (gcc version 5.2.1 20151010 (Debian 5.2.1-22) ) #1 SMP PREEMPT Wed Oct 28 18:14:25 UTC 2015 Command line: BOOT_IMAGE=/vmlinuz-4.2.5-zgsrv20080 root=/dev/mapper/aida-aida_root ro console=ttyS0,115200n8 net.ifnames=1 KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls x86/fpu: Legacy x87 FPU detected. x86/fpu: Using 'lazy' FPU context switches. e820: BIOS-provided physical RAM map: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved BIOS-e820: [mem 0x0000000000100000-0x00000000df14dfff] usable BIOS-e820: [mem 0x00000000df14e000-0x00000000dfffffff] reserved BIOS-e820: [mem 0x00000000f8000000-0x00000000f8ffffff] reserved BIOS-e820: [mem 0x0000000100000000-0x000000011effffff] usable NX (Execute Disable) protection: active SMBIOS 2.7 present. e820: last_pfn = 0x11f000 max_arch_pfn = 0x400000000 x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT e820: last_pfn = 0xdf14e max_arch_pfn = 0x400000000 found SMP MP-table at [mem 0x000f25b0-0x000f25bf] mapped at [ffff8800000f25b0] Using GB pages for direct mapping init_memory_mapping: [mem 0x00000000-0x000fffff] init_memory_mapping: [mem 0x11ee00000-0x11effffff] init_memory_mapping: [mem 0x100000000-0x11edfffff] init_memory_mapping: [mem 0xc0000000-0xdf14dfff] init_memory_mapping: [mem 0x00100000-0xbfffffff] RAMDISK: [mem 0x371a0000-0x378c7fff] ACPI: Early table checksum verification disabled ACPI: RSDP 0x00000000000F2590 000014 (v00 CORE ) ACPI: RSDT 0x00000000DF162424 000044 (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: FACP 0x00000000DF164CE8 0000F4 (v04 CORE COREBOOT 00000000 CORE 00000000) ACPI: DSDT 0x00000000DF1624C8 0027DA (v02 PCENG COREBOOT 00010001 INTL 20121114) ACPI: FACS 0x00000000DF164CA8 000040 ACPI: SPCR 0x00000000DF164DDC 000050 (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: HPET 0x00000000DF164E30 000038 (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: APIC 0x00000000DF164E68 00005C (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: HEST 0x00000000DF164EC8 000198 (v01 CORE COREBOOT 00000000 CORE 00000000) ACPI: SSDT 0x00000000DF165060 00168E (v02 AMD ALIB 00000001 MSFT 04000000) ACPI: SSDT 0x00000000DF1666F0 00030C (v01 AMD POWERNOW 00000001 AMD 00000001) ACPI: SSDT 0x00000000DF166A00 000045 (v02 CORE COREBOOT 0000002A CORE 0000002A) No NUMA configuration found Faking a node at [mem 0x0000000000000000-0x000000011effffff] NODE_DATA(0) allocated [mem 0x11effc000-0x11effffff] Zone ranges: DMA [mem 0x0000000000001000-0x0000000000ffffff] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] Normal [mem 0x0000000100000000-0x000000011effffff] Movable zone start for each node Early memory node ranges node 0: [mem 0x0000000000001000-0x000000000009efff] node 0: [mem 0x0000000000100000-0x00000000df14dfff] node 0: [mem 0x0000000100000000-0x000000011effffff] Initmem setup node 0 [mem 0x0000000000001000-0x000000011effffff] ACPI: PM-Timer IO Port: 0x808 IOAPIC[0]: apic_id 2, version 33, address 0xfec00000, GSI 0-23 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level) Using ACPI (MADT) for SMP configuration information ACPI: HPET id: 0x43538210 base: 0xfed00000 smpboot: Allowing 2 CPUs, 0 hotplug CPUs e820: [mem 0xe0000000-0xf7ffffff] available for PCI devices Booting paravirtualized kernel on bare hardware clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:2 nr_node_ids:1 PERCPU: Embedded 31 pages/cpu @ffff88011ec00000 s89432 r8192 d29352 u1048576 Built 1 zonelists in Node order, mobility grouping on. Total pages: 1024337 Policy zone: Normal Kernel command line: BOOT_IMAGE=/vmlinuz-4.2.5-zgsrv20080 root=/dev/mapper/aida-aida_root ro console=ttyS0,115200n8 net.ifnames=1 PID hash table entries: 4096 (order: 3, 32768 bytes) Memory: 4014848K/4162480K available (4613K kernel code, 600K rwdata, 1656K rodata, 916K init, 452K bss, 147632K reserved, 0K cma-reserved) Preemptible hierarchical RCU implementation. Build-time adjustment of leaf fanout to 64. RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2. RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2 NR_IRQS:4352 nr_irqs:440 16 Console: colour dummy device 80x25 console [ttyS0] enabled clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns tsc: Fast TSC calibration using PIT tsc: Detected 1000.033 MHz processor Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.06 BogoMIPS (lpj=4000132) pid_max: default: 32768 minimum: 301 ACPI: Core revision 20150619 ACPI: All ACPI Tables successfully acquired Security Framework initialized Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) Initializing cgroup subsys blkio Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys net_prio CPU: Physical Processor ID: 0 CPU: Processor Core ID: 0 mce: CPU supports 6 MCE banks Last level iTLB entries: 4KB 512, 2MB 8, 4MB 4 Last level dTLB entries: 4KB 512, 2MB 8, 4MB 4, 1GB 0 Freeing SMP alternatives memory: 20K (ffffffff8179f000 - ffffffff817a4000) ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0 ..MP-BIOS bug: 8254 timer not connected to IO-APIC ...trying to set up timer (IRQ0) through the 8259A ... ..... (found apic 0 pin 0) ... ....... failed. ...trying to set up timer as Virtual Wire IRQ... ..... works. smpboot: CPU0: AMD G-T40E Processor (fam: 14, model: 02, stepping: 00) Performance Events: AMD PMU driver. ... version: 0 ... bit width: 48 ... generic registers: 4 ... value mask: 0000ffffffffffff ... max period: 00007fffffffffff ... fixed-purpose events: 0 ... event mask: 000000000000000f NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. x86: Booting SMP configuration: .... node #0, CPUs: #1 x86: Booted up 1 node, 2 CPUs smpboot: Total of 2 processors activated (4000.13 BogoMIPS) BUG: unable to handle kernel NULL pointer dereference at (null) IP: [< (null)>] (null) PGD 0 Oops: 0010 [#1] PREEMPT SMP Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.2.5-zgsrv20080 #1 Hardware name: PC Engines APU, BIOS SageBios_PCEngines_APU-45 04/05/2014 task: ffff880119ab0b00 ti: ffff880119ab4000 task.ti: ffff880119ab4000 RIP: 0010:[<0000000000000000>] [< (null)>] (null) RSP: 0018:ffff880119ab7ed0 EFLAGS: 00010246 RAX: ffff88011a005458 RBX: 0000000000000000 RCX: ffffffff816afce0 RDX: 0000000000000000 RSI: ffff88011a005458 RDI: ffff88011a005408 RBP: 0000000000000002 R08: 0000000000000000 R09: ffff880119c00008 R10: 0000000000000000 R11: 00000000000000b4 R12: ffffffff817d6e20 R13: ffff88011a005408 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88011ec00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000000 CR3: 000000000162f000 CR4: 00000000000006f0 Stack: ffffffff816e3d01 0000000000000001 00000000007a1308 000000000000a040 0000000000000001 0000000000000000 ffffffff816e0a16 0000000000000001 0000000000000000 000000000000008b ffffffff816d0fa7 ffffffff81474aa0 Call Trace: [<ffffffff816e3d01>] ? setup_ioapic_dest+0xcb/0xe5 [<ffffffff816e0a16>] ? native_smp_cpus_done+0xa0/0xa8 [<ffffffff816d0fa7>] ? kernel_init_freeable+0xd7/0x1e3 [<ffffffff81474aa0>] ? rest_init+0x90/0x90 [<ffffffff81474aa9>] ? kernel_init+0x9/0xe0 [<ffffffff8147d6df>] ? ret_from_fork+0x3f/0x70 [<ffffffff81474aa0>] ? rest_init+0x90/0x90 Code: Bad RIP value. RIP [< (null)>] (null) RSP <ffff880119ab7ed0> CR2: 0000000000000000 ---[ end trace 613f6ef92efea625 ]--- Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009 I haven't debugged a Linux kernel in years, how do I proceed from here? Is there a possibility that the "bad" commit can be identified without a bisect orgy? The APU is rather productive for my LAN and me rebooting it many times will not please my wife ;-) Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 -- To unsubscribe from this list: send the line "unsubscribe stable" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html