On Fri, Oct 16, 2015 at 3:43 AM, Frank Michel <fmfranck@xxxxxxxxx> wrote: > Hello, > > We are encountering a backup problem using tgtd. > > Though, we believe it to be a kernel problem, you might have > encountered such an issue before and have some lights about it. > > We backup a Linux RHEL 6 system using an iSCSI link connected to an > usb3 drive on another system. The problem is that the backup process > on the source system is writing too fast for the network to send data. > > The number of dirty pages of unwritten data keep rising on the source > system because of the network bottleneck.This keep growing until the > source system freeze because of too much allocated memory. That sounds likely. I often encounter similar issues, for example using rsync to copy a lot of data from one usb drive to another. Very very annoying. > > We believe it’s a kernel issue which has not been fixed even in the > most recent kernel versions (3.3). To fix that, it would require a max > dirty pages per device kernel parameter which is not available at this > moment. > > Have you heard of any « strategy » which we could use as a workaround > for this specific problem ? Since you say it is the pagecache that completely fills up, I assume you are mounting the remote iSCSI device locally ? I don't have any great strategies, but you could try : * mount the filesystem using 'sync' mount option. It MIGHT prevent the buildup of dirty pages. * the tool you use to copy/write the files to the backup direcotry, does it support O_DIRECT ? That might allow writing to the destination completely bypassing the page cache. Both these suggestions are probably likely to have a HUGE negative impact on performance when accessing a block device across the network. :-( Maybe that is not critical in a backup application ? Or, you could also do this, it is very likely to guarantee to avoid this issue, but it means that what you store on the remote USB device is no longer just a copy of the directory structure : * Use TGTD as a virtual tape library and do backups to "virtual tape" instead. Since this is no longer a block device as far as the kernel is concerned it will not consume any pagecache. Also, since tape access is designed to be in streaming mode you will likely still have very high throughput when writing your backup. I.e. you will not have the performance issues from above. TGTD supports VTL emulation , yey! If you are willing to do "virtual tape" I would recommend that you set your USB device to keep as many xxxGB files as possible. The command to create and initialize a file to be used as a "virtual tape" is tgtimg. This document describes the basics on how to set TGTD up as a virtual tape library : https://github.com/fujita/tgt/blob/master/doc/README.vtl Since VTL means you will have to talk to the virtual media changer to switch tapes, you can use the mtx tool that should be available on every linux distro. It you rather want a simple but nice GUI to swap/load/unload tapes, you can use this: https://github.com/rosjat/mtx-gui regards ronnie sahlberg > > Thanks for your help, > -- > To unsubscribe from this list: send the line "unsubscribe stgt" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe stgt" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
