On Tue, Mar 03, 2020 at 09:50:00AM -0700, David Brown wrote: > I am using xfsdump with multiple levels to backup my main system > (Fedora f31, xfs root and home on LVM). Kernel is > 5.5.6-201.fc31.x86_64. > > When doing a test restore of my backup, when I reach my level 2 > backup, I get the following warnings: > > xfsrestore: directory post-processing > xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory The orphanage is a special xfsrestore thing. files and directories without any tree links at the time they are restored from the media file are placed into the orphanage. When a link to that object in the orphanage is found, it moves it into place. However, directory post processing also removes stale tree entries. Hence you can get files in the orphanage that have no directories that point to them at all, and they then get discarded because there is nowhere to restore them to. This process can be recursive, as directory post processing can move entries to the orphanage and vice versa, meaning that files that were skipped in the restore because they had not tree linkage now have their parent directory in place and so can now be restored. There's a chicken-and-egg problem in this algorithm, too, which can lead to an unresolvable recursion loop, so restore may not be able to resolve everything correctly form such a media file.... > In addition, I get hundreds of these. They seem to all be related to > that same directory. > > xfsrestore: restoring non-directory files > xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/cmake_install.cmake failed: No such file or directory: discarding ino 2833282 > xfsrestore: WARNING: open of orphanage/422178422.2232121414/modules/atmel/asf/common/cmake_install.cmake failed: No such file or directory: discarding ino 2833283 Yep, this is typical of a dump taken on a live, changing filesystem. i.e. This is not an xfsrestore problem but a result of changing filesystem structure while xfsdump is running. That is, xfsdump has scanned the filesystem, found all the files/dirs that need to be backed up, then gone and read all the data into the media file. It does a separate directory traversal to write the directory heirarchy into the dump inventory. As a result of the time time difference between the file data backup vs the directory struct backup, the dump media file contains files that don't exist in the directory tree and the directory tree can contain files that don't exist in the dump. Hence the backup data in the media file can be inconsistent as it is not a point in time snapshot of the live filesystem, and the only time you are going to see that inconsistency is whey you go to restore the media file. > When I verify the restore (using https://github.com/d3zd3z/rsure), I > indeed find that <<path>>/b-cloud is missing, but another directory > next to it is still present that should be removed. Yup, that can happen on a live filesystem backup. If you want a guaranteed consistent backup, then you need to run xfsdump on a static snapshot of the filesystem and not the live filesystem itself. > Restores of subsequent print similar messages (and don't restore the > directory), until one of them dies with this: > > xfsrestore: WARNING: unable to rename dir orphanage/422178422.2232121414 to dir <<path>>/b-cloud: No such file or directory > xfsrestore: node.c:539: node_map: Assertion `nh != NH_NULL' failed. > > Any ideas on how to debug this? The node freeing code has been passed a null pointer, which indicates there's an inconsistency in the internal directory tree that restore is building. Smells like a double-free is occuring. The usual is in order: turn the debugging logging all the way up, run it under gdb to find the path that leads to the crash, make your eyeballs bleed by reading the xfsdump code... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
