I forgot to send this to the list. -------- Ursprungligt meddelande -------- Ämne: Re: Broken nilfs2 filesystem Datum: Fri, 26 Jul 2013 18:49:40 +0200 Från: Anton Eliasson <devel@xxxxxxxxxxxxxxxx> Till: Vyacheslav Dubeyko <slava@xxxxxxxxxxx> Vyacheslav Dubeyko skrev 2013-07-26 14:37:
Hi Anton, Do you ready to try to obtain debug output? I am really waiting your readiness because your opportunity to reproduce the issue is very important. I think that it needs to enable such configuration options: 1. CONFIG_NILFS2_DEBUG_SHOW_ERRORS 2. CONFIG_NILFS2_DEBUG_BASE_OPERATIONS 3. CONFIG_NILFS2_DEBUG_MDT_FILES 4. CONFIG_NILFS2_DEBUG_SEGMENTS_SUBSYSTEM 5. CONFIG_NILFS2_DEBUG_BLOCK_MAPPING 6. CONFIG_NILFS2_DEBUG_DUMP_STACK Thanks, Vyacheslav Dubeyko.
Hi, Thanks for the reminder! I tried it out a few weeks ago. My plan then was to restore the old root system (running Linux 3.9) and /home to the mechanical hard drive, boot them up and use that system to build the newer kernel with your kernel. This would make minimal changes to the system I had experienced the bugs with, I thought. Building the kernel took me some time but I eventually succeded. I forgot those configuration options though so I suppose that build did nothing out of the ordinary, even though it had the patches. I got it to boot but then ran in to the issue of incompatible video drivers. I've been down that road before and I did not feel like going back. My next plan was to wait until 3.10 got released into the core repository which happened yesterday or so. I would then use my fully updated system (currently running the 3.10 kernel on an ext4 root filesystem) to build a kernel with the debug patches. It would be a slightly different system with many updated packages, but at least the root filesystem will be a healthy ext4. So that's what I did today. Buildlogs excerpts are at the bottom of this message. I got the kernel to boot and X to start. But as soon as anything tried to read from /home (even just logging into a virtual terminal as a regular user), that terminal froze. Logging in as root to a different VT showed that syslog-ng and nilfs_cleanerd took 100% CPU each. After a while the entire system froze. Pressing Alt+SysRq+R caused the kernel to spew out an endless stream of call traces to the terminal, leaving no other way to reboot than a hard reset. After a reboot I found that the running nilfs_cleanerd does not respond to either a SIGTERM or a SIGKILL. Changing the mount options of /home from rw,noatime,discard to ro,norecovery,noatime,discard seems to work. No crashes during login to a VT. Read-only is no fun though. Next attempt is rw,nogc,noatime,discard. It seems to work also, though there are a lot of call traces in dmesg. After just a few reboots and a few minutes of uptime, /var/log/kernel.log, messages.log and everything.log have grown to 3.5 GB each. My / is only 30 GB so I can't sustain this for very long. I have aborted the experiments for today. kernel.log has 35 million lines and compresses to 220 MB. I've uploaded it here (http://antoneliasson.se/publicdump/kernel.log.20130726.gz). What should I do next? Build logs ======= * Download kernel package build scripts and patches (from the Arch Build System). Unpack and apply downstream patches: $ makepkg -o ==> Skapar paket: linux 3.10.2-1 (fre jul 26 17:08:16 CEST 2013) ==> Retrieving sources... -> Hittade linux-3.10.tar.xz -> Laddar ner patch-3.10.2.xz... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 178 100 178 0 0 188 0 --:--:-- --:--:-- --:--:-- 188 100 27876 100 27876 0 0 13761 0 0:00:02 0:00:02 --:--:-- 37978 -> Hittade config -> Hittade config.x86_64 -> Hittade linux.preset -> Hittade change-default-console-loglevel.patch ==> Validerar källfiler med md5sums... linux-3.10.tar.xz ... Godkänd patch-3.10.2.xz ... Godkänd config ... Godkänd config.x86_64 ... Godkänd linux.preset ... Godkänd change-default-console-loglevel.patch ... Godkänd ==> Extracting sources... -> Extraherar linux-3.10.tar.xz med bsdtar -> Extraherar patch-3.10.2.xz med xz ==> Startar prepare()... patching file Documentation/parisc/registers patching file MAINTAINERS patching file Makefile patching file arch/arm/boot/dts/imx23.dtsi patching file arch/arm/boot/dts/imx28.dtsi patching file arch/arm/boot/dts/imx6dl.dtsi patching file arch/arm/boot/dts/imx6q.dtsi patching file arch/arm/include/asm/mmu_context.h patching file arch/arm/kernel/perf_event.c [...] patching file mm/page_alloc.c patching file mm/slab.c patching file net/ceph/auth_none.c patching file kernel/printk.c Hunk #1 succeeded at 56 with fuzz 2 (offset -2 lines). ==> Källor är redo. $ ls -l -rw-r--r-- 1 anton anton 2,7K 26 jul 00.06 alsa-firmware-loading-3.8.8.patch -rw-r--r-- 1 anton anton 605 26 jul 00.06 change-default-console-loglevel.patch -rw-r--r-- 1 anton anton 142K 26 jul 17.13 config -rw-r--r-- 1 anton anton 142K 26 jul 00.06 config~ -rw-r--r-- 1 anton anton 138K 26 jul 17.13 config.x86_64 -rw-r--r-- 1 anton anton 138K 26 jul 00.06 config.x86_64~ -rw-r--r-- 1 anton anton 926 26 jul 00.06 linux.install -rw-r--r-- 1 anton anton 376 26 jul 00.06 linux.preset drwxr-xr-x 2 anton anton 4,0K 26 jul 17.06 nilfs2-debug-output-patch-set-25-06-2013 -rw-r--r-- 1 anton anton 13K 26 jul 00.06 PKGBUILD drwxr-xr-x 3 anton anton 4,0K 26 jul 17.08 src * Apply nilfs2 debug patches: $ cd src/linux-3.10/ $ for file in ../../nilfs2-debug-output-patch-set-25-06-2013/*.patch; do patch -p1 < "$file"; done patching file fs/nilfs2/Kconfig patching file fs/nilfs2/debug.h patching file fs/nilfs2/Kconfig patching file fs/nilfs2/debug.h patching file fs/nilfs2/dir.c patching file fs/nilfs2/file.c patching file fs/nilfs2/inode.c patching file fs/nilfs2/ioctl.c patching file fs/nilfs2/namei.c patching file fs/nilfs2/nilfs.h [...] patching file fs/nilfs2/segbuf.c patching file fs/nilfs2/segment.c patching file fs/nilfs2/sufile.c patching file fs/nilfs2/super.c patching file fs/nilfs2/the_nilfs.c patching file fs/nilfs2/Kconfig patching file fs/nilfs2/debug.h * Append the following lines to config (just in case) and config.x86_64 (which I assume I will use): CONFIG_NILFS2_DEBUG_SHOW_ERRORS=y CONFIG_NILFS2_DEBUG_BASE_OPERATIONS=y CONFIG_NILFS2_DEBUG_MDT_FILES=y CONFIG_NILFS2_DEBUG_SEGMENTS_SUBSYSTEM=y CONFIG_NILFS2_DEBUG_BLOCK_MAPPING=y CONFIG_NILFS2_DEBUG_DUMP_STACK=y * Build the kernel. There were some interactive configuration options concerning nilfs2 that I responded yes to: $ makepkg -e ==> Skapar paket: linux 3.10.2-1 (fre jul 26 17:16:41 CEST 2013) ==> Checking runtime dependencies... ==> Checking buildtime dependencies... ==> VARNING: Using existing src/ tree ==> Startar build()... HOSTCC scripts/basic/fixdep HOSTCC scripts/kconfig/conf.o SHIPPED scripts/kconfig/zconf.tab.c SHIPPED scripts/kconfig/zconf.lex.c SHIPPED scripts/kconfig/zconf.hash.c HOSTCC scripts/kconfig/zconf.tab.o HOSTLD scripts/kconfig/conf scripts/kconfig/conf --silentoldconfig Kconfig * * Restart config... * * * File systems * [...] NILFS2 file system support (NILFS2_FS) [M/n/y/?] m NILFS2 debugging (NILFS2_DEBUG) [N/y/?] (NEW) y Use pr_debug() instead of printk() (NILFS2_USE_PR_DEBUG) [N/y/?] (NEW) y Show internal errors (NILFS2_DEBUG_SHOW_ERRORS) [N/y/?] (NEW) y Enable dump stack output (NILFS2_DEBUG_DUMP_STACK) [N/y/?] (NEW) y [...] # # configuration written to .config # SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_32.h SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_64.h SYSHDR arch/x86/syscalls/../include/generated/uapi/asm/unistd_x32.h SYSTBL arch/x86/syscalls/../include/generated/asm/syscalls_32.h SYSHDR arch/x86/syscalls/../include/generated/asm/unistd_32_ia32.h SYSHDR arch/x86/syscalls/../include/generated/asm/unistd_64_x32.h SYSTBL arch/x86/syscalls/../include/generated/asm/syscalls_64.h WRAP arch/x86/include/generated/asm/clkdev.h CHK include/generated/uapi/linux/version.h UPD include/generated/uapi/linux/version.h CHK include/generated/utsrelease.h UPD include/generated/utsrelease.h HOSTCC arch/x86/tools/relocs_32.o [...] INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/edgeport/down3.bin INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/keyspan_pda/keyspan_pda.fw INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/keyspan_pda/xircom_pgs.fw INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/cpia2/stv0672_vp4.bin INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/yam/1200.bin INSTALL /home/anton/build/linux/pkg/linux/lib/firmware/yam/9600.bin DEPMOD 3.10.2-1-ARCH ==> Städar upp efter installationen... -> Rensar oönskade filer... -> Komprimerar man och info sidor... ==> Creating package "linux"... -> Skapar .PKGINFO fil... -> Lägger till install fil... -> Generating .MTREE file... -> Komprimerar paket... ==> Startar package_linux-headers()... ==> Städar upp efter installationen... -> Rensar oönskade filer... -> Komprimerar man och info sidor... ==> Creating package "linux-headers"... -> Skapar .PKGINFO fil... -> Generating .MTREE file... -> Komprimerar paket... ==> Startar package_linux-docs()... ==> Städar upp efter installationen... -> Rensar oönskade filer... -> Komprimerar man och info sidor... ==> Creating package "linux-docs"... -> Skapar .PKGINFO fil... -> Generating .MTREE file... -> Komprimerar paket... ==> Leaving fakeroot environment. ==> Kompilering klar: linux 3.10.2-1 (fre jul 26 17:46:42 CEST 2013) * Install over the stock 3.10 kernel, point /home to the old nilfs2 filesystem in /etc/fstab and reboot. -- Best Regards, Anton Eliasson -- To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
