> On Jul 14, 2024, at 12:33 PM, BIPUL KHAN <bipul364@xxxxxxxxx> wrote: > > Hello, > We have developed and run a large-scale mobile financial system using a microservice architecture, which includes 110 databases (big and small). We use PostgreSQL as the primary database inside Kubernetes via the CrunchyData operator. > Now I would like to share my problem. Our data storage has grown significantly since 2020, and we are using Ceph (rook-Ceph). We now have 21 TB of data. > Our current situation is critical as we urgently need to move our data center. Our source and destination data centers have a 1 GB/s data transfer rate. We attempted to add a new Ceph node with the same cluster, which started data rebalancing, but it’s taking a significant amount of time. This directly and severely impacts our primary transactions, which involve reading and writing to the database. > What is the best option for moving data from one data center to a new one? We would appreciate your expertise. Would you happento know how to move it any other way? > It is mentioned that the two different data centers are different Kubernetes clusters. > I would appreciate some ideas. Feel free to let me know if you need any other adjustments! > Let me know if there are any further changes you'd like! Plan to migrate off Ceph as soon as possible. It is simply not suited for this--yes, I am speaking from experience. The rebalancing will hammer you when you can least afford it. WekaFS, Pure Storage, heck NFS. Or node-local storage, XFS or ZFS. Really, just about anything else. (It might help if you clarify whether "move our data center" really means what I assumed, that you own hardware and are moving it, or whether you're going to another cloud provider. You're probably going to have to migrate your databases one at a time--even if you temporarily upgraded your bandwidth to 10Gbps you'd have too long of a down time. So, one at a time, set up a standby streaming replica, then when they're all populated and up-to-date, take a very brief downtime (seconds) to switch over. This assumes that 1Gbps can actually handle your stream of normal updates with enough room left over to be transferring the base files. (Current versions of Crunchy support the notion of cross-cluster replication.)
