> -------- Forwarded Message -------- > Subject: Fio Checksum tracking and enhanced trim workloads > Date: Sun, 7 May 2017 23:54:16 -0400 > From: paul houlihan <phoulihan9@xxxxxxxxx> > To: fio@xxxxxxxxxxxxxxx, Jens Axboe <axboe@xxxxxxxxx> > > I have a submission for fio that enhances the data corruption detection and diagnosis capabilities taking fio from pretty good corruption detection to absolute guarantees. I would like these changes on the tracking branch???? to be reviewed and considered for inclusion to fio. A quick review would be helpful as I am losing access to test systems shortly. > > > These changes were used by a Virtual Machine caching company to assure data integrity. Most testing was on Linux 64 bits and windows 32/64 bits. The windows build still had an issue with compile time asserts in libfio.c that I worked around by commenting out the asserts as this looked like a performance restriction. This should be researched more. The initial development was on version fio 2.2.10 sources and I just ported the changes to fio latest sources and tested on linux but haven’t yet test on windows. No testing on all other fio supported OSes was done, although the changes are almost exclusively to OS independent code. > > > The absolute guarantees are brought about by tracking checksums to prevent a stale but intact prior version of a block being returned and by verifying all reads. I was surprised to learn about the number of times fio performed concurrent I/O to the same blocks which yields indeterminate results that prevent data integrity verification. Thus a number of options are not supported when tracking is enabled. Sounds interesting! Are the patches available on Github or otherwise published? I have a couple of patches related to overlapping I/Os (see https://github.com/axboe/fio/pull/343 and https://github.com/sitsofe/fio/commit/6b4cfeb95fca2d75a291c54ca20162470c837a38 ) because I can't afford for such I/O to be sent to the storage. Perhaps it would be possible to amend your work to detect the overlaps in a more efficient manner? How does this work play with varying blocksizes (bsrange etc)? > These limitations can be addressed by setting the verify_track option which > > allocates a memory array to track the header and data checksums to assure > > data integrity is absolute. The array starts out empty at the beginning of > > each fio job and is filled in as reads or writes occur, once defined the > > checksums from succeeding I/Os must all match. This option extends checksum > > verification to all reads in all workloads, not just the read-only workloads. Are you sure? I thought readwrite verifying workloads caused the reads to be verifying if the block had been written in the same workload? > fio_header.py - a python script that displays the contents of the block header > > in a corrupt data file. Many would find this useful. Perhaps it could be ported to C like t/verify-state.c ? > Here are the related parameter descriptions from HOWTO: > > > option verify_track=bool > > > Fio normally verifies data within a verify_intervalwith checksums and file > > offsets embedded in the data. However a prior version of a block could be > > returned and verified successfully. When verify_track is enabled the checksum > > for every verify_interval in the file is stored in a table and all read data > > must match the checksums in the table. The tracking table is sized as > > (size / verify_interval) * 4 bytes. For very large size= option settings, > > such a large memory allocation may impact testing. Reads assume that the entire > > file has been previously written with a verification format using the same > > verify_interval. When verify_track is enabled, all reads are verified, whether > > writes are present in the workload or not. Sharing files by threads within a job > > is supported but not between jobs running concurrently so use the stonewall > > option when more than one non-global job is present. Verify of trimmed blocks > > is described for the verify_track_trim_zero option. When disabled, fio falls > > back on verification described under the verify option. The restrictions when > > enabling the verify_track option are: > > - randommap is required > > - softrandommap is not supported > > - lfsr random generator not supported when using multiple block sizes > > - stonewall option required when more than one job present > > - file size must be an even multiple of the block size when iodepth > 1 > > - verify_backlog not supported when iodepth > 1 > > - verify_async is not supported > > - file sharing between concurrent jobs not supported > > - numjobs must be 1 > > - io_submit_mode must be set to "inline" Yeah I've noticed a number of issues trying to verify with io_submit_mode=offload ... > option verify_track_log=bool > > > If set when verify_track is set then on a clean shutdown, fio writes the checksum > > for each data block that has been read or written to a log named > > (datafilename).tracking.log. If set when fio reopens this data file and a tracking > > log exists then the checksums are read into the tracking table and used to validate > > every subsequent read. This allows rigorous validation of data integrity as data > > files are passed between fio jobs or over the termination of fio and restart on > > the same system or on another system or after an OS reboot. Reverting a virtual > > machine to a snapshot can be tested by saving the tracking log after a successful > > fio run and later restoring the saved log after reverting the virtual machine. > > The log is deleted after being read in, so on abnormal termination no stale > > checksums can be used. This option, the data file size and verify_interval > > parameters should not change between jobs in the same run or on restart of fio. > > Defaults to off. verify_track_dir defines the tracking log's directory. How does this interact with verify_state_* (http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-verify-state-load ) ? -- Sitsofe | http://sucs.org/~sits/ -- To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
