issues with replicating data to a new brick

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello everybody,

I have some kind of a situation here

I want to move some volumes to new hosts. the idea is to add the new
bricks to the volume, sync and then drop the old bricks.

starting point is:


Volume Name: Server_Monthly_02
Type: Replicate
Volume ID: 0ada8e12-15f7-42e9-9da3-2734b04e04e9
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: chastcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Brick2: chglbcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Options Reconfigured:
features.scrub: Inactive
features.bitrot: off
nfs.disable: on
auth.allow: 127.0.0.1,10.30.28.43,10.30.28.44,10.30.28.17,10.30.28.18,10.8.13.132,10.30.28.30,10.30.28.31
performance.readdir-ahead: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on


root@chastcvtprd04:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/";
SUPPORT_URL="http://help.ubuntu.com/";
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/";
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
root@chastcvtprd04:~# uname -a
Linux chastcvtprd04 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9
19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@chastcvtprd04:~# dpkg -l | grep gluster
ii  glusterfs-client                     3.8.15-ubuntu1~xenial1
               amd64        clustered file-system (client package)
ii  glusterfs-common                     3.8.15-ubuntu1~xenial1
               amd64        GlusterFS common libraries and translator
modules
ii  glusterfs-server                     3.8.15-ubuntu1~xenial1
               amd64        clustered file-system (server package)



root@chastcvtprd04:~# df -h  /data/glusterfs/Server_Monthly/2I-1-40/brick
Filesystem      Size  Used Avail Use% Mounted on
/dev/bcache47   7.3T  7.3T   45G 100% /data/glusterfs/Server_Monthly/2I-1-40


then I add the new brick


Volume Name: Server_Monthly_02
Type: Replicate
Volume ID: 0ada8e12-15f7-42e9-9da3-2734b04e04e9
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: chastcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Brick2: chglbcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Brick3: chglbglsprd02:/data/glusterfs/Server_Monthly/1I-1-51/brick
Options Reconfigured:
features.scrub: Inactive
features.bitrot: off
nfs.disable: on
auth.allow: 127.0.0.1,10.30.28.43,10.30.28.44,10.30.28.17,10.30.28.18,10.8.13.132,10.30.28.30,10.30.28.31
performance.readdir-ahead: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on


root@chglbglsprd02:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/";
SUPPORT_URL="http://help.ubuntu.com/";
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/";
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
root@chglbglsprd02:~# uname -a
Linux chglbglsprd02 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@chglbglsprd02:~# dpkg -l | grep gluster
ii  glusterfs-client                     3.8.15-ubuntu1~xenial1
             amd64        clustered file-system (client package)
ii  glusterfs-common                     3.8.15-ubuntu1~xenial1
             amd64        GlusterFS common libraries and translator
modules
ii  glusterfs-server                     3.8.15-ubuntu1~xenial1
             amd64        clustered file-system (server package)


then healing kicks in and the cluster starts copying data to the new brick
unfortunately after a while it starts complaining

[2018-04-10 14:39:32.057443] E [MSGID: 113072]
[posix.c:3457:posix_writev] 0-Server_Monthly_02-posix: write failed:
offset 0, [No space left on device]
[2018-04-10 14:39:32.057538] E [MSGID: 115067]
[server-rpc-fops.c:1346:server_writev_cbk] 0-Server_Monthly_02-server:
22835126: WRITEV 0 (48949669-ba1c-4735-b83c-71340f1bb64f) ==> (No
space left on device) [No space left on device]


root@chglbglsprd02:~# df -h /data/glusterfs/Server_Monthly/1I-1-51/brick
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdaq       7.3T  7.3T   20K 100% /data/glusterfs/Server_Monthly/1I-1-51



there's no other I/O going on on this volume, so the copy process
should be straight forward
BUT I noticed that there are a lot of sparse files on this volume



Any ideas on how to make it work?
If you need more details, please let me known and I'll try to make
them available


Kind Regards
Bernhard
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux