Just some more informations :
I use Ceph Octopus 15.2.13 on Ubuntu 18.04, deployed with ceph-ansible,
no container (except grafana/prometheus/node_exporter).
Le 2021-07-12 23:42, Gilles Mocellin a écrit :
Hello Cephers,
I'm disappointed. I thought I'v had found a good way to migrate from
one data pool to another, without too much downtime.
I use XFS on RBD, via KRBD to store backups (see another thread). XFS
with reflink and crc (accelerate Veeam merges).
Also, I want to migrate from a EC k3m2 to EC k8m2 on 13 nodes / 130
osds.
I cannot use rbd migration without downtime, due to krbd. Even, I
tried and it was really slow.
But I saw the layering capacity offered by cloning, and thought of
this method :
- unmount filesystem / unmap rbd device
- take snapshot / protect it
- clone snapshot to new image, changing data pool
- rename old image, new image for the new image ti have the name of
the original one
- map new image and mount filesystem
This is fast, because it's only COW.
Starting from here, I thought that all new write will be on the new
data pool.
And waiting for the backup retentions, I can migrate without doing
anything.
To finalize, when there will not be too much data to move, I would do
a rbd flatten, and be able to delete the source image on old data
pool.
But...
I have problems.
On one image, my backups constantly fail, or a retry is triggered. No
clear message, but I/O seems slow, perhaps timeouts.
On another image, I had XFS kernel errors (on metadata) and the
filesystem shuts down during the night (and backups).
Jul 10 01:32:28 fidcl-mrs4-vbr-repo-02 kernel: [6108174.156744] XFS
(rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr
0x8a2d4a4f8 len 8 error 5
Jul 10 01:32:47 fidcl-mrs4-vbr-repo-02 kernel: [6108193.510924] XFS
(rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr
0x8a2d4a4f8 len 8 error 5
Jul 10 01:32:58 fidcl-mrs4-vbr-repo-02 kernel: [6108204.696929] XFS
(rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr
0x8a2d4a4f8 len 8 error 5
Jul 10 01:33:13 fidcl-mrs4-vbr-repo-02 kernel: [6108219.857228] XFS
(rbd1): metadata I/O error in "xfs_buf_iodone_callback_error" at daddr
0x8a2d4a4f8 len 8 error 5
I unmount it and try to remount without success. xfs_repairs tells me
I had to mount it to replay journal, and if I cannot, ignore it and...
loose data.
Last unattended thing. On one image that is still mounted, and seems
to work, I tries to launch a flatten operation, to see how long it can
last and if it manage to finish, if my backups are doing better on it.
But her are the output I have, thought it seems to continue...
Image flatten: 2% complete...2021-07-12T23:22:55.998+0200 7f511a7fc700
-1 librbd::operation::FlattenRequest: 0x7f50fc000f20 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 0% complete...2021-07-12T23:23:32.142+0200 7f5119ffb700
-1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete:
encountered error: (85) Interrupted system call should be restarted
2021-07-12T23:23:47.382+0200 7f5119ffb700 -1
librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 2% complete...2021-07-12T23:23:58.926+0200 7f5119ffb700
-1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete:
encountered error: (85) Interrupted system call should be restarted
Image flatten: 0% complete...2021-07-12T23:24:01.318+0200 7f5119ffb700
-1 librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete:
encountered error: (85) Interrupted system call should be restarted
2021-07-12T23:24:07.422+0200 7f5119ffb700 -1
librbd::operation::FlattenRequest: 0x7f50fc0015d0 should_complete:
encountered error: (85) Interrupted system call should be restarted
So, either I it bugs, or layering, cloning, flattening is not supposed
to work in my context... Perhaps due to changing data pool ? Erasure
Coding ?
I'm now stuck.
I have ~400To of data to move from an EC3+2 to EC8+2 pool, and I'm
only seeing one solution : stopping my backups during the copy, that
will last weeks...
(No, I can't stay on EC3+2, I'v sold to my management and my
colleagues that we'll have near 1PB usable on that cluster).
Thanx for reading, if you're still there !
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
