contents of /proc/mdstat root@superior:/mnt/backup# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: <none> root@superior:/mnt/backup# Here are the steps I ran (minus some mounting other devices and looking around for mdadm tracks on the old os disk) 410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo /dev/{1}) 411 apt install parallel 412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo /dev/{1}) 413 echo $DEVICES 414 cat /proc/partitions 415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg 416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg" 417 echo $DEVICES 418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES 419 ls /dev/loop* 420 dc 421 cd /mnt/backup/ 422 ls 423 parallel truncate -s300G overlay-{/} ::: $DEVICES 424 ls 425 ls -la 426 df -h 427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES 428 ls /dev/mapper/ 429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES) 430 echo $OVERLAYS 431 dmsetup status 432 mdadm --assemble --force /dev/md1 $OVERLAYS 433 history 434 dmsetup status 435 echo $OVERLAYS 436 mdadm --assemble --force /dev/md0 $OVERLAYS 437 cat /proc/partitions 438 mkdir /mnt/oldroot << look for inird mdadm files >> 484 echo $OVERLAYS 485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg << cancelled out of 485, review instructions... >> 486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb /dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf /dev/mapper/sdg 487 fsck -n /dev/md0 488 mdadm --stop /dev/md0 489 echo $DEVICES 490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES 491 dmsetup status 492 ls 493 rm overlay-* 494 ls 495 parallel losetup -d ::: /dev/loop[0-9]* 496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES 497 parallel truncate -s300G overlay-{/} ::: $DEVICES 498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES 499 dmsetup status 500 /sbin/reboot 501 history 502 dmsetup status 503 mount 504 cat /proc/partitions 505 nano /etc/fstab 506 mount /mnt/backup/ 507 ls /mnt/backup/ 508 rm /mnt/backup/ 509 rm /mnt/backup/overlay-sd* 510 emacs setupOverlay & 511 ps auxww | grep emacs 512 kill 65017 513 ls /dev/loo* 514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg' 515 echo $DEVICES 516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7 {#}' ::: $DEVICES 517 ls /dev/loo* 518 parallel truncate -s4000G overlay-{/} ::: $DEVICES 519 ls 520 rm overlay-sd* 521 cd /mnt/bak 522 cd /mnt/backup/ 523 ls 524 parallel truncate -s4000G overlay-{/} ::: $DEVICES 525 ls -la 526 blockdev --getsize /dev/sdb 527 man losetup 528 man losetup 529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f --show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup create {/}' ::: $DEVICES 530 dmsetup status 531 history | grep mdadm 532 history 533 dmsetup status 534 history | grep dmsetup 535 dmsetup status 536 dmsetup remove sdg 537 dmsetup ls --tree 538 lsof 539 dmsetup ls --tre 540 dmsetup ls --tree 541 lsof | grep -i sdg 542 lsof | grep -i sdf 543 history |grep dmsetup | less 544 dmsetup status 545 history > ~plsander/Documents/raidIssues/joblog On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@xxxxxxxxxxx> wrote: > > >>>>> "Peter" == Peter Sanders <plsander@xxxxxxxxx> writes: > > > encountering a puzzling situation. > > dmsetup is failing to return. > > I don't think you need to use dmsetup in your case, but can you post > *all* the commands you ran before you got to this point, and the > output of > > cat /proc/mdstat > > as well? Thinking on this some more, you might need to actually also > add: > > --assume-clean > > to the 'mdadm create ....' string, since you don't want it to zero the > array or anything. > > Sorry for not remembering this at the time! > > So if you can, please just start over from scratch, showing the setup > of the loop devices, the overlayfs setup, and the building the RAID6 > array, along with the cat /proc/mdstat after you do the initial build. > > John > > P.S. For those who hated my email citing tool, I pulled it out for > now. Only citing with > now. :-) > > > root@superior:/mnt/backup# dmsetup status > > sdg: 0 5860533168 snapshot 16/8388608000 16 > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > sde: 0 5860533168 snapshot 16/8388608000 16 > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > dmsetup remove sdg runs for hours. > > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list. > > > dmsetup status shows: > > sdf: 0 5860533168 snapshot 16/8388608000 16 > > sde: 0 5860533168 snapshot 16/8388608000 16 > > sdd: 0 5860533168 snapshot 16/8388608000 16 > > sdc: 0 5860533168 snapshot 16/8388608000 16 > > sdb: 0 5860533168 snapshot 16/8388608000 16 > > > dmsetup ls --tree > > root@superior:/mnt/backup# dmsetup ls --tree > > sdf (253:3) > > ├─ (7:3) > > └─ (8:80) > > sde (253:1) > > ├─ (7:1) > > └─ (8:64) > > sdd (253:2) > > ├─ (7:2) > > └─ (8:48) > > sdc (253:0) > > ├─ (7:0) > > └─ (8:32) > > sdb (253:5) > > ├─ (7:5) > > └─ (8:16) > > > any suggestions? > > > > > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@xxxxxxxxxxxxxxx> wrote: > >> > >> On 30/08/2022 14:27, Peter Sanders wrote: > >> > > >> > And the victory conditions would be a mountable file system that passes a fsck? > >> > >> Yes. Just make sure you delve through the file system a bit and satisfy > >> yourself it looks good, too ... > >> > >> Cheers, > >> Wol