[PATCH 00/32] qemu: backup: Rewrite/simplify internals and document semantics

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Simplify how we approach bitmap handling to increase robustness and
understanding of the code. Additionally document intricacies of how we
approach handling of bitmaps so that in cases when users decide to
do snapshots outside of libvirt they can re-create the bitmaps so that
they will still work with backups (requires new qemu-img).

I'll also follow up documenting implicationsof blockjobs on bitmaps
later.

To fetch this including a commit which allows testing:

 git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-rewrite-and-document

This doesn't require any special qemu patches.

Note that this shares many patches from my previous posting which tried
to make use of 'block-dirty-bitmap-populate', but keeps explicit bitmap
tracking for now. We'll be able to revisit use of the blockjob if the
qemu work will continue somehow.

Peter Krempa (32):
  qemu: backup: Split up code traversing checkpoint list looking for
    bitmaps
  qemu: backup: Fix backup of disk skipped in an intermediate checkpoint
  conf: backup: Store incremental backup checkpoint name per-disk
  qemu: backup: Move fetching of checkpoint list for incremental backup
  qemublocktest: Add 'empty' test case for bitmaps
  qemublocktest: Add 'empty' case for incremental backup test
  qemublocktest: Add 'empty' case for checkpoint deletion
  qemublocktest: Add 'empty' case for blockcopy bitmap handling test
  qemublocktest: Add 'empty' case for checkpoint bitmap handling
  qemublocktest: Disable testcases for all bitmap handling
  qemublocktest: Delete 'synthetic' bitmap test cases
  qemublocktest: Extract printing of nodename list
  qemu: checkpoint: Don't chain bitmaps for checkpoints
  qemublocktest: Replace 'basic' bitmap detection test case data
  qemublocktest: Replace 'snapshots' bitmap detection test case data
  qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps
  qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots'
    cases
  qemublocktest: Add new 'synthetic' bitmap detection and validation
    test case
  qemu: checkpoint: Don't merge checkpoints during deletion
  qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to
    TEST_CHECKPOINT_DELETE
  qemublocktest: Re-introduce testing of checkpoint deletion
  qemu: block: Add universal helper for merging dirty bitmaps for all
    scenarios
  qemu: backup: Rewrite backup bitmap handling to the new bitmap
    semantics
  qemublocktest: Add 'basic' tests for backup bitmap handling
  qemublocktest: Add 'snapshots' tests for backup bitmap handling
  qemu: Rewrite bitmap handling for block commit
  qemublocktest: Add 'basic' tests for commit bitmap handling
  qemublocktest: Add 'snapshots' tests for block commit bitmap handling
  qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job
    private data
  qemu: Rewrite bitmap handling for block copy
  qemublocktest: Add test cases for handling bitmaps during block-copy
  kbase: Add document outlining internals of incremental backup in qemu

 docs/kbase.html.in                            |    3 +
 docs/kbase/incrementalbackupinternals.rst     |  210 +++
 src/conf/backup_conf.c                        |    8 +
 src/conf/backup_conf.h                        |    1 +
 src/qemu/qemu_backup.c                        |  199 +--
 src/qemu/qemu_backup.h                        |   12 +-
 src/qemu/qemu_block.c                         |  500 +++---
 src/qemu/qemu_block.h                         |   21 +-
 src/qemu/qemu_blockjob.c                      |  109 +-
 src/qemu/qemu_blockjob.h                      |    3 -
 src/qemu/qemu_checkpoint.c                    |  174 +--
 src/qemu/qemu_checkpoint.h                    |    1 -
 src/qemu/qemu_domain.c                        |   22 -
 src/qemu/qemu_driver.c                        |   70 +-
 tests/qemublocktest.c                         |  254 ++-
 .../backupmerge/basic-deep-out.json           |   35 +-
 .../backupmerge/basic-flat-out.json           |   23 +-
 .../backupmerge/basic-intermediate-out.json   |   23 +-
 .../backupmerge/empty-out.json                |    3 +
 .../backupmerge/snapshot-deep-out.json        |   38 -
 .../backupmerge/snapshot-flat-out.json        |    6 -
 .../snapshot-intermediate-out.json            |   14 -
 .../backupmerge/snapshots-deep-out.json       |   41 +
 .../backupmerge/snapshots-flat-out.json       |   25 +
 .../snapshots-intermediate-out.json           |   29 +
 tests/qemublocktestdata/bitmap/basic.json     |  229 +--
 tests/qemublocktestdata/bitmap/basic.out      |    8 +-
 tests/qemublocktestdata/bitmap/empty.json     |   70 +
 tests/qemublocktestdata/bitmap/empty.out      |    1 +
 .../bitmap/snapshots-synthetic-broken.json    |  837 ----------
 .../bitmap/snapshots-synthetic-broken.out     |   14 -
 .../snapshots-synthetic-checkpoint.json       |  827 ----------
 .../bitmap/snapshots-synthetic-checkpoint.out |   13 -
 tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++----------
 tests/qemublocktestdata/bitmap/snapshots.out  |   13 +-
 tests/qemublocktestdata/bitmap/synthetic.json |  606 ++++++--
 tests/qemublocktestdata/bitmap/synthetic.out  |   19 +-
 .../bitmapblockcommit/basic-1-2               |   54 +-
 .../bitmapblockcommit/basic-1-3               |   54 +-
 .../bitmapblockcommit/basic-2-3               |    1 -
 .../qemublocktestdata/bitmapblockcommit/empty |    1 +
 .../bitmapblockcommit/snapshots-1-2           |   76 +-
 .../bitmapblockcommit/snapshots-1-3           |   77 +-
 .../bitmapblockcommit/snapshots-1-4           |   99 +-
 .../bitmapblockcommit/snapshots-1-5           |  103 +-
 .../bitmapblockcommit/snapshots-2-3           |   48 +-
 .../bitmapblockcommit/snapshots-2-4           |   58 +-
 .../bitmapblockcommit/snapshots-2-5           |   62 +-
 .../bitmapblockcommit/snapshots-3-4           |   34 +-
 .../bitmapblockcommit/snapshots-3-5           |   42 +-
 .../bitmapblockcommit/snapshots-4-5           |   17 -
 .../snapshots-synthetic-broken-1-2            |   57 -
 .../snapshots-synthetic-broken-1-3            |  112 --
 .../snapshots-synthetic-broken-1-4            |  119 --
 .../snapshots-synthetic-broken-1-5            |  119 --
 .../snapshots-synthetic-broken-2-3            |   89 --
 .../snapshots-synthetic-broken-2-4            |   96 --
 .../snapshots-synthetic-broken-2-5            |   96 --
 .../snapshots-synthetic-broken-3-4            |   27 -
 .../snapshots-synthetic-broken-3-5            |   27 -
 .../snapshots-synthetic-broken-4-5            |   20 -
 .../bitmapblockcopy/basic-deep-out.json       |   53 +-
 .../bitmapblockcopy/basic-shallow-out.json    |   53 +-
 .../bitmapblockcopy/empty-deep-out.json       |    0
 .../bitmapblockcopy/empty-shallow-out.json    |    0
 .../bitmapblockcopy/snapshots-deep-out.json   |   95 +-
 .../snapshots-shallow-out.json                |   98 +-
 .../checkpointdelete/basic-current-out.json   |   20 -
 .../basic-intermediate1-out.json              |   13 -
 .../basic-intermediate2-out.json              |   13 -
 .../basic-intermediate3-out.json              |   13 -
 .../checkpointdelete/empty-out.json           |    1 +
 .../snapshots-current-out.json                |   20 -
 .../snapshots-intermediate1-out.json          |   20 +-
 .../snapshots-intermediate2-out.json          |   42 +-
 .../snapshots-intermediate3-out.json          |   43 -
 .../snapshots-noparent-out.json               |   15 +
 ...hots-synthetic-checkpoint-current-out.json |   29 -
 ...ynthetic-checkpoint-intermediate1-out.json |   31 -
 ...ynthetic-checkpoint-intermediate2-out.json |   34 -
 ...ynthetic-checkpoint-intermediate3-out.json |   61 -
 ...ots-synthetic-checkpoint-noparent-out.json |   27 -
 .../synthetic-current-out.json                |    9 +
 .../synthetic-intermediate1-out.json          |   11 +
 .../synthetic-intermediate2-out.json          |   11 +
 .../synthetic-intermediate3-out.json          |   19 +
 .../synthetic-noparent-out.json               |   11 +
 .../blockjob-blockdev-in.xml                  |    4 -
 88 files changed, 2927 insertions(+), 5150 deletions(-)
 create mode 100644 docs/kbase/incrementalbackupinternals.rst
 create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json
 delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json
 delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json
 delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json
 create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json
 create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json
 create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json
 create mode 100644 tests/qemublocktestdata/bitmap/empty.json
 create mode 100644 tests/qemublocktestdata/bitmap/empty.out
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out
 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5
 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json
 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json

-- 
2.26.2




[Index of Archives]     [Virt Tools]     [Libvirt Users]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [KDE Users]     [Fedora Tools]

  Powered by Linux