Hello Cephers, I'm disappointed. I thought I'v had found a good way to migrate from one data pool to another, without too much downtime. I use XFS on RBD, via KRBD to store backups (see another thread). XFS with reflink and crc (accelerate Veeam merges). Also, I want to migrate from a EC k3m2 to EC k8m2 on 13 nodes / 130 osds. I cannot use rbd migration without downtime, due to krbd. Even, I tried and it was really slow. But I saw the layering capacity offered by cloning, and thought of this method : - unmount filesystem / unmap rbd device - take snapshot / protect it - clone snapshot to new image, changing data pool - rename old image, new image for the new image ti have the name of the original one - map new image and mount filesystem This is fast, because it's only COW. Starting from here, I thought that all new write will be on the new data pool. And waiting for the backup retentions, I can migrate without doing anything. To finalize, when there will not be too much data to move, I would do a rbd flatten, and be able to delete the source image on old data pool. But... I have problems. On one image, my backups constantly fail, or a retry is triggered. No clear message, but I/ O seems slow, perhaps timeouts. On another image, I had XFS kernel errors (on metadata) and the filesystem shuts down during the night (and backups). Jul 10 01:32:28 fidcl-mrs4-vbr-repo-02 kernel: [6108174.156744] XFS (rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x8a2d4a4f8 len 8 error 5 Jul 10 01:32:47 fidcl-mrs4-vbr-repo-02 kernel: [6108193.510924] XFS (rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x8a2d4a4f8 len 8 error 5 Jul 10 01:32:58 fidcl-mrs4-vbr-repo-02 kernel: [6108204.696929] XFS (rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x8a2d4a4f8 len 8 error 5 Jul 10 01:33:13 fidcl-mrs4-vbr-repo-02 kernel: [6108219.857228] XFS (rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr 0x8a2d4a4f8 len 8 error 5 I unmount it and try to remount without success. xfs_repairs tells me I had to mount it to replay journal, and if I cannot, ignore it and... loose data. Last unattended thing. On one image that is still mounted, and seems to work, I tries to launch a flatten operation, to see how long it can last and if it manage to finish, if my backups are doing better on it. But her are the output I have, thought it seems to continue... Image flatten: 2% complete...2021-07-12T23:22:55.998+0200 7f511a7fc700 -1 librbd::operation::FlattenRequest: 0x7f50fc000f20 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 0% complete...2021-07-12T23:23:32.142+0200 7f5119ffb700 -1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete: encountered error: (85) Interrupted system call should be restarted 2021-07-12T23:23:47.382+0200 7f5119ffb700 -1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 2% complete...2021-07-12T23:23:58.926+0200 7f5119ffb700 -1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete: encountered error: (85) Interrupted system call should be restarted Image flatten: 0% complete...2021-07-12T23:24:01.318+0200 7f5119ffb700 -1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete: encountered error: (85) Interrupted system call should be restarted 2021-07-12T23:24:07.422+0200 7f5119ffb700 -1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete: encountered error: (85) Interrupted system call should be restarted So, either I it bugs, or layering, cloning, flattening is not supposed to work in my context... Perhaps due to changing data pool ? Erasure Coding ? I'm now stuck. I have ~400To of data to move from an EC3+2 to EC8+2 pool, and I'm only seeing one solution : stopping my backups during the copy, that will last weeks... (No, I can't stay on EC3+2, I'v sold to my management and my colleagues that we'll have near 1PB usable on that cluster). Thanx for reading, if you're still there ! _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |