[RFC PATCH 00/41] qemu: Rewrite checkpoint code for 'block-dirty-bitmap-populate'

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Use 'block-dirty-bitmap-populate' and change how we create bitmaps
corresponding to checkpoints to simplify the code and also properly
integrate with backing chain images created outside of libvirt.

This patchset changes how we approach checkpoints by keeping one bitmap
per checkpoint and disk and not propagating the bitmaps into overlays on
snapshots. This massively simplifies the code handling all the
operations during blockjobs and backups.

While the change isn't compatible with checkpoints created previously,
we didn't yet enable the support for checkpoints/backups.

Note that 'block-dirty-bitmap-populate' is _not_ in finished state in
qemu yet, so I'm posting this as RFC and as reference for qemu
developers of the usefulnes of it.


The changes can be fetched by:

  git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-bitmap-populate

Note that the above branch also contains a commit enabling incremental
backup for simpler testing.

I've pushed the appropriate qemu patches for convenience here:

  git fetch https://gitlab.com/pipo.sk/qemu.git block-dirty-bitmap-populate

Peter Krempa (41):
  virErrorPreserveLast: Return the saved error object in addition to
    storing it
  util: error: Introduce VIR_ERROR_AUTOPRESERVE_LAST macro
  qemuBackupDiskDataCleanup: Use VIR_ERROR_AUTO_PRESERVE_LAST
  qemu: backup: Split up code traversing checkpoint list looking for
    bitmaps
  qemu: backup: Fix backup of disk skipped in an intermediate checkpoint
  conf: backup: Store incremental backup checkpoint name per-disk
  qemu: backup: Move fetching of checkpoint list for incremental backup
  qemublocktest: Add 'empty' test case for bitmaps
  qemublocktest: Add 'empty' case for incremental backup test
  qemublocktest: Add 'empty' case for checkpoint deletion
  qemublocktest: Add 'empty' case for blockcopy bitmap handling test
  qemublocktest: Add 'empty' case for checkpoint bitmap handling
  qemublocktest: Disable testcases for all bitmap handling
  qemublocktest: Delete 'synthetic' bitmap test cases
  qemublocktest: Extract printing of nodename list
  qemu: checkpoint: Don't chain bitmaps for checkpoints
  qemublocktest: Replace 'basic' bitmap detection test case data
  qemu: snapshot: Don't propagate bitmaps to upper layers
  qemublocktest: Replace 'snapshots' bitmap detection test case data
  qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps
  qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots'
    cases
  qemublocktest: Add new 'synthetic' bitmap detection and validation
    test case
  qemu: checkpoint: Don't merge checkpoints during deletion
  qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to
    TEST_CHECKPOINT_DELETE
  qemublocktest: Re-introduce testing of checkpoint deletion
  qemucapabilitiestest: Bump qemu-5.1 caps on x86_64 for
    'dirty-bitmap-populate'
  qemu: monitor: Add support for 'block-dirty-bitmap-populate'
    transaction member
  qemuDomainStorageSourcePrivate: Add per-source private blockjob
  qemu: blockjob: Introduce 'populate' blockjob
  qemu: domain: Introduce helper for always fetching virStorageSource
    private data
  qemu: block: Add helper to add temporary block bitmaps from allocation
    maps
  qemu: block: Add universal helper for merging dirty bitmaps for all
    scenarios
  qemu: backup: Rewrite backup bitmap handling to the new bitmap
    semantics
  qemublocktest: Add 'basic' tests for backup bitmap handling
  qemublocktest: Add 'snapshots' tests for backup bitmap handling
  qemu: Rewrite bitmap handling for block commit
  qemublocktest: Add 'basic' tests for commit bitmap handling
  qemublocktest: Add 'snapshots' tests for block commit bitmap handling
  qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job
    private data
  qemu: Rewrite bitmap handling for block copy
  qemublocktest: Add test cases for handling bitmaps during block-copy

 src/conf/backup_conf.c                        |    8 +
 src/conf/backup_conf.h                        |    1 +
 src/qemu/qemu_backup.c                        |  221 +-
 src/qemu/qemu_backup.h                        |   13 +-
 src/qemu/qemu_block.c                         |  674 ++--
 src/qemu/qemu_block.h                         |   35 +-
 src/qemu/qemu_blockjob.c                      |  182 +-
 src/qemu/qemu_blockjob.h                      |   16 +-
 src/qemu/qemu_checkpoint.c                    |  174 +-
 src/qemu/qemu_checkpoint.h                    |    1 -
 src/qemu/qemu_domain.c                        |   46 +-
 src/qemu/qemu_domain.h                        |    6 +
 src/qemu/qemu_driver.c                        |  112 +-
 src/qemu/qemu_monitor.c                       |   11 +
 src/qemu/qemu_monitor.h                       |    7 +
 src/qemu/qemu_monitor_json.c                  |   18 +
 src/qemu/qemu_monitor_json.h                  |    6 +
 src/util/virerror.c                           |   18 +-
 src/util/virerror.h                           |   12 +-
 tests/qemublocktest.c                         |  267 +-
 .../backupmerge/basic-deep-out.json           |   35 +-
 .../backupmerge/basic-flat-out.json           |   23 +-
 .../backupmerge/basic-intermediate-out.json   |   23 +-
 .../backupmerge/empty-out.json                |    3 +
 .../backupmerge/snapshot-deep-out.json        |   38 -
 .../backupmerge/snapshot-flat-out.json        |    6 -
 .../snapshot-intermediate-out.json            |   14 -
 .../backupmerge/snapshots-deep-out.json       |   46 +
 .../backupmerge/snapshots-flat-out.json       |   25 +
 .../snapshots-intermediate-out.json           |   31 +
 tests/qemublocktestdata/bitmap/basic.json     |  229 +-
 tests/qemublocktestdata/bitmap/basic.out      |    8 +-
 tests/qemublocktestdata/bitmap/empty.json     |   70 +
 tests/qemublocktestdata/bitmap/empty.out      |    1 +
 .../bitmap/snapshots-synthetic-broken.json    |  837 -----
 .../bitmap/snapshots-synthetic-broken.out     |   14 -
 .../snapshots-synthetic-checkpoint.json       |  827 -----
 .../bitmap/snapshots-synthetic-checkpoint.out |   13 -
 tests/qemublocktestdata/bitmap/snapshots.json | 1254 +++-----
 tests/qemublocktestdata/bitmap/snapshots.out  |    6 +-
 tests/qemublocktestdata/bitmap/synthetic.json |  606 +++-
 tests/qemublocktestdata/bitmap/synthetic.out  |   19 +-
 .../bitmapblockcommit/basic-1-2               |   66 +-
 .../bitmapblockcommit/basic-1-3               |   66 +-
 .../bitmapblockcommit/basic-2-3               |    1 -
 .../qemublocktestdata/bitmapblockcommit/empty |    1 +
 .../bitmapblockcommit/snapshots-1-2           |   24 +-
 .../bitmapblockcommit/snapshots-1-3           |   42 +-
 .../bitmapblockcommit/snapshots-1-4           |   97 +-
 .../bitmapblockcommit/snapshots-1-5           |   99 +-
 .../bitmapblockcommit/snapshots-2-3           |   23 -
 .../bitmapblockcommit/snapshots-2-4           |   55 +-
 .../bitmapblockcommit/snapshots-2-5           |   59 +-
 .../bitmapblockcommit/snapshots-3-4           |   25 +-
 .../bitmapblockcommit/snapshots-3-5           |   29 +-
 .../bitmapblockcommit/snapshots-4-5           |   32 -
 .../snapshots-synthetic-broken-1-2            |   57 -
 .../snapshots-synthetic-broken-1-3            |  112 -
 .../snapshots-synthetic-broken-1-4            |  119 -
 .../snapshots-synthetic-broken-1-5            |  119 -
 .../snapshots-synthetic-broken-2-3            |   89 -
 .../snapshots-synthetic-broken-2-4            |   96 -
 .../snapshots-synthetic-broken-2-5            |   96 -
 .../snapshots-synthetic-broken-3-4            |   27 -
 .../snapshots-synthetic-broken-3-5            |   27 -
 .../snapshots-synthetic-broken-4-5            |   20 -
 .../bitmapblockcopy/basic-deep-out.json       |   65 +-
 .../bitmapblockcopy/basic-shallow-out.json    |   65 +-
 .../bitmapblockcopy/empty-deep-out.json       |    0
 .../bitmapblockcopy/empty-shallow-out.json    |    0
 .../bitmapblockcopy/snapshots-deep-out.json   |  104 +-
 .../snapshots-shallow-out.json                |   24 +-
 .../checkpointdelete/basic-current-out.json   |   20 -
 .../basic-intermediate1-out.json              |   13 -
 .../basic-intermediate2-out.json              |   13 -
 .../basic-intermediate3-out.json              |   13 -
 .../checkpointdelete/empty-out.json           |    1 +
 .../snapshots-current-out.json                |   20 -
 .../snapshots-intermediate1-out.json          |   13 -
 .../snapshots-intermediate2-out.json          |   51 -
 .../snapshots-intermediate3-out.json          |   50 -
 .../snapshots-noparent-out.json               |   16 -
 ...hots-synthetic-checkpoint-current-out.json |   29 -
 ...ynthetic-checkpoint-intermediate1-out.json |   31 -
 ...ynthetic-checkpoint-intermediate2-out.json |   34 -
 ...ynthetic-checkpoint-intermediate3-out.json |   61 -
 ...ots-synthetic-checkpoint-noparent-out.json |   27 -
 .../synthetic-current-out.json                |    9 +
 .../synthetic-intermediate1-out.json          |   11 +
 .../synthetic-intermediate2-out.json          |   11 +
 .../synthetic-intermediate3-out.json          |   19 +
 .../synthetic-noparent-out.json               |   11 +
 .../caps_5.1.0.x86_64.replies                 | 2737 +++++++++--------
 .../caps_5.1.0.x86_64.xml                     |    2 +-
 tests/qemumonitorjsontest.c                   |    3 +-
 .../blockjob-blockdev-in.xml                  |    4 -
 96 files changed, 4077 insertions(+), 6717 deletions(-)
 create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json
 delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json
 delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json
 delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json
 create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json
 create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json
 create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json
 create mode 100644 tests/qemublocktestdata/bitmap/empty.json
 create mode 100644 tests/qemublocktestdata/bitmap/empty.out
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json
 delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out
 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5
 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5
 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json
 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json
 delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json
 create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json

-- 
2.26.2




[Index of Archives]     [Virt Tools]     [Libvirt Users]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [KDE Users]     [Fedora Tools]

  Powered by Linux