Rsync fails on local transfers

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I've got a script I wrote up which backs up my computer using rsync.  A
short overview is that I created a file system image using dd, and now I
mount the image and rsync from the server's HD to the image file.

Periodically during large backups, I get the following error:

    rsync: connection unexpectedly closed (16931 bytes received so far)
    [sender]
    rsync error: error in rsync protocol data stream (code 12) at
    io.c(453) [sender=2.6.9]

When I Google for the problem, I get many results about a SSH connection
timing out because of inactivity during the delete stage, but I'm not
using SSH, so I don't get why I would have this problem: both the source
and destination are local.

Any ideas on how to debug this would be great. I've thought of using
strace, but it'd result in a huge file which would have to be sorted
through (as rsync isn't technically crashing, rsync would continue with
a shutdown procedure and the logging wouldn't stop right at the point of
failure).

Attached is the backup script I use (the configuration file it loads in
is very simple, so I'm not including that).

Justin W

#!/bin/bash

# User where the reports of the backup are sent to
MAILTO=root
# Path to backup's configuration file
CONF_PATH=/root

# Overview of backup process
#  1) Read and parse configuration file
#  2) Loop through each backup
#  3) Create a FS snapshot using LVM
#  4) Mount the snapshot FS
#  5) Mount our backup file in a loop filesystem (we can guarantee the space
#     will be present this way, and it allows selective recovery of files if
#     need be)
#  6) rsync the differences between the snapshot FS and the backup FS
#  7) Unmount the backup FS
#  8) Unmount the snapshot FS
#  9) Delete the snapshot LVM (leaving it will degrade performance)
# 10) Repeat 3-9 if necessary

# Give us a large enough path to execute commands
PATH=/bin:/sbin:/usr/bin:/usr/sbin

# Save the old IFS variable so that we can restore it when we're done
ORIGIFS=$IFS
IFS=$(echo -en "\n\b")

[ -f "$CONF_PATH/backup.conf" ] || {
	echo "Configuration file not found ($CONF_PATH/backup.conf). Exiting.";
	exit 1; }

# Read from the configuration file any exclude filters for the rsync output
LOG_EXCLUDE=$(cat "$CONF_PATH/backup.conf" | egrep "^[ \t]*LOG_EXCLUDE=" |
              cut -d'=' -f 2-)
LOG_FILE=/tmp/backup$$.log

# Get each line from the configuration file one at a time, ignoring anything
# that starts with a pound (#) sign
for i in $(cat "$CONF_PATH/backup.conf" | egrep -v "^[ \t]*#.*|^[ \t]*LOG_EXCLUDE=.*"); do
	# Parse out the locations we need
	VOL=$(echo $i | sed "s@^\([^:]*\):\(.*\)@\1@")
	TARG=$(echo $i | sed "s@^\([^:]*\):\(.*\)@\2@")

	# Check to make sure the paths are valid
	[ -b $VOL ]  || { echo "Source volume not valid: $VOL. Skipping";
	                  continue; }
	[ -f $TARG ] || { echo "Target image not valid: $TARG. Skipping";
	                  continue; }

	# Get necessary information about the source volume.
	# Get only the necessary line, and not the headers, and then compress the
	# info so it is only separated by a single space (makes finding fields
	# easier in the next steps). There will be an extra space at the beginning, 
	# so fields are 1 greater than what they seem.
	#
	# Make sure the volume is a LVM Logical Volume
	lvs -a $VOL 2>/dev/null 1>/dev/null || {
		echo "$VOL not an LVM Logical Volume ($?). Skipping."; continue; }
	# Now we actually collect the info
	INFO=$(lvs -a $VOL | tail -n 1 | sed 's/[ ][ ]*/ /g')
	# Field 1: Logical Volume Name
	VOL_NAME=$(echo $INFO | cut -d' ' -f 2)
	# Field 2: Volume Group Name
	GROUP_NAME=$(echo $INFO | cut -d' ' -f 3)
	# Field 3: Attributes - Not needed right now, but may come in handy
	ATTRS=$(echo $INFO | cut -d' ' -f 4)
	# Field 4: Volume Size
	VOL_SIZE=$(echo $INFO | cut -d' ' -f 5)

	# Now we just create a name for the snapshot volume, and then we can
	# create it. Use a unique name guaranteed to be the same through multiple
	# runs (or else blkid.tab fills with unneccessary partition info)
	SNAP_NAME=backup_$(echo "$GROUP_NAME/$VOL_NAME" | md5sum | cut -c-32)
	SNAP_PATH=/dev/${GROUP_NAME}/${SNAP_NAME}
	lvcreate -s -L${VOL_SIZE} -n $SNAP_NAME $VOL >/dev/null || {
		echo "Snapshot of '${GROUP_NAME}/${SNAP_NAME}' failed ($?). Skipping.";
		continue; }
	
	# Mount our filesystems at temporary mount locations
	SNAP_MOUNT=/tmp/$SNAP_NAME
	TARG_MOUNT=/tmp/target-$SNAP_NAME
	mkdir $SNAP_MOUNT || {
		echo "Failed to create snapshot mount point ($?). Skipping.";
		cleanup; continue; }
	mkdir $TARG_MOUNT || {
		echo "Failed to create target mount point ($?). Skipping.";
		cleanup; continue; }
	mount $SNAP_PATH $SNAP_MOUNT || {
		echo "Mounting snapshot failed ($?). Skipping.";
		cleanup; continue; }
	mount -o loop $TARG $TARG_MOUNT || {
		echo "Mounting target image failed ($?). Skipping.";
		cleanup; continue; }

	# Update the backup file system with files from the snapshot
	# IMPORTANT!!: The trailing slash on the source is necessary for it to
	# work in the expected way, or else all backups will end up within a
	# subdirectory with the name $SNAP_MOUNT
	strace -o /tmp/backup-fs \
	rsync -aHEAXxv --stats --delete --ignore-errors $SNAP_MOUNT/ $TARG_MOUNT | \
		egrep -v "$LOG_EXCLUDE" >> $LOG_FILE

	# Cleanup stuff is put into a function so that it can be called before
	# an exit call
	function cleanup() {
		# Unmount the backups and remove the temporary dirs
		# Errors may occur here, but we don't really care. Just pipe the errors
		# to /dev/null
		umount $SNAP_MOUNT 2>/dev/null 1>/dev/null
		umount $TARG_MOUNT 2>/dev/null 1>/dev/null
		[ -d $SNAP_MOUNT ] && rmdir $SNAP_MOUNT
		[ -d $TARG_MOUNT ] && rmdir $TARG_MOUNT

		# Delete the snapshot
		[ -b $SNAP_PATH ] && lvremove -f $SNAP_PATH >/dev/null
	}
	# Now call our cleanup routine on normal completions
	cleanup
done

if [ -f $LOG_FILE ]; then
	# Mail the log file, and then delete the file
	mail -s "Backups on $(hostname)" $MAILTO < $LOG_FILE
	rm $LOG_FILE
fi

# Restore the IFS
IFS=$ORIGIFS

-- 
fedora-list mailing list
fedora-list@xxxxxxxxxx
To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list
[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [Fedora Magazine]     [Fedora News]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Maintainers]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [Fedora Fonts]     [ATA RAID]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [SSH]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Tux]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Asterisk PBX]     [Fedora Sparc]     [Fedora Universal Network Connector]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux