Re: Getting glusterfs to expand volume size to brick size

[Artem Russakovskii <archon810@xxxxxxxxx>]

Hi Nithya,

I'm on Gluster 4.0.1. 

I don't think the bricks were smaller before - if they were, maybe 20GB because Linode's minimum is 20GB, then I extended them to 25GB, resized with resize2fs as instructed, and rebooted many times over since. Yet, gluster refuses to see the full disk size.

Here's the status detail output:

gluster volume status dev_apkmirror_data detail

Status of volume: dev_apkmirror_data

------------------------------------------------------------------------------

Brick                : Brick pylon:/mnt/pylon_block1/dev_apkmirror_data

TCP Port             : 49152               

RDMA Port            : 0                   

Online               : Y                   

Pid                  : 1263                

File System          : ext4                

Device               : /dev/sdd            

Mount Options        : rw,relatime,data="">

Inode Size           : 256                 

Disk Space Free      : 23.0GB              

Total Disk Space     : 24.5GB              

Inode Count          : 1638400             

Free Inodes          : 1625429             

------------------------------------------------------------------------------

Brick                : Brick pylon:/mnt/pylon_block2/dev_apkmirror_data

TCP Port             : 49153               

RDMA Port            : 0                   

Online               : Y                   

Pid                  : 1288                

File System          : ext4                

Device               : /dev/sdc            

Mount Options        : rw,relatime,data="">

Inode Size           : 256                 

Disk Space Free      : 24.0GB              

Total Disk Space     : 25.5GB              

Inode Count          : 1703936             

Free Inodes          : 1690965             

------------------------------------------------------------------------------

Brick                : Brick pylon:/mnt/pylon_block3/dev_apkmirror_data

TCP Port             : 49154               

RDMA Port            : 0                   

Online               : Y                   

Pid                  : 1313                

File System          : ext4                

Device               : /dev/sde            

Mount Options        : rw,relatime,data="">

Inode Size           : 256                 

Disk Space Free      : 23.0GB              

Total Disk Space     : 24.5GB              

Inode Count          : 1638400             

Free Inodes          : 1625433  

What's interesting here is that the gluster volume size is exactly 1/3 of the total (8357M * 3 = 25071M). Yet, each block device is separate, and the total storage available is 25071M on each brick.

The fstab is as follows:

/dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1 ext4 defaults 0 2

/dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2 ext4 defaults 0 2

/dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3 ext4 defaults 0 2

localhost:/dev_apkmirror_data    /mnt/dev_apkmirror_data1   glusterfs defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0

localhost:/dev_apkmirror_data    /mnt/dev_apkmirror_data2   glusterfs defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0

localhost:/dev_apkmirror_data    /mnt/dev_apkmirror_data3   glusterfs defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0

localhost:/dev_apkmirror_data    /mnt/dev_apkmirror_data_ganesha   nfs4 defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 0 0

The latter entry is for an nfs ganesha test, in case it matters (which, btw, fails miserably with all kinds of stability issues about broken pipes).

Note: this is a test server, so all 3 bricks are attached and mounted on the same server.

Sincerely,
Artem

--
Founder, Android Police, APK Mirror, Illogical Robot LLC

beerpla.net | +ArtemRussakovskii | @ArtemR

On Sun, Apr 15, 2018 at 10:56 PM, Nithya Balachandran <nbalacha@xxxxxxxxxx> wrote:

What version of Gluster are you running? Were the bricks smaller earlier?

Regards,

Nithya

On 15 April 2018 at 00:09, Artem Russakovskii <archon810@xxxxxxxxx> wrote:

Hi,

I have a 3-brick replicate volume, but for some reason I can't get it to expand to the size of the bricks. The bricks are 25GB, but even after multiple gluster restarts and remounts, the volume is only about 8GB.

I believed I could always extend the bricks (we're using Linode block storage, which allows extending block devices after they're created), and gluster would see the newly available space and extend to use it.

Multiple Google searches, and I'm still nowhere. Any ideas?

df | ack "block|data"

Filesystem                                                   1M-blocks     Used Available Use% Mounted on

/dev/sdd                                                        25071M    1491M    22284M   7% /mnt/pylon_block1

/dev/sdc                                                        26079M    1491M    23241M   7% /mnt/pylon_block2

/dev/sde                                                        25071M    1491M    22315M   7% /mnt/pylon_block3

localhost:/dev_apkmirror_data                                    8357M     581M     7428M   8% /mnt/dev_apkmirror_data1

localhost:/dev_apkmirror_data                                    8357M     581M     7428M   8% /mnt/dev_apkmirror_data2

localhost:/dev_apkmirror_data                                    8357M     581M     7428M   8% /mnt/dev_apkmirror_data3

gluster volume info

 

Volume Name: dev_apkmirror_data

Type: Replicate

Volume ID: cd5621ee-7fab-401b-b720-08863717ed56

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: pylon:/mnt/pylon_block1/dev_apkmirror_data

Brick2: pylon:/mnt/pylon_block2/dev_apkmirror_data

Brick3: pylon:/mnt/pylon_block3/dev_apkmirror_data

Options Reconfigured:

disperse.eager-lock: off

cluster.lookup-unhashed: auto

cluster.read-hash-mode: 0

performance.strict-o-direct: on

cluster.shd-max-threads: 12

performance.nl-cache-timeout: 600

performance.nl-cache: on

cluster.quorum-count: 1

cluster.quorum-type: fixed

network.ping-timeout: 5

network.remote-dio: enable

performance.rda-cache-limit: 256MB

performance.parallel-readdir: on

network.inode-lru-limit: 500000

performance.md-cache-timeout: 600

performance.cache-invalidation: on

performance.stat-prefetch: on

features.cache-invalidation-timeout: 600

features.cache-invalidation: on

performance.io-thread-count: 32

server.event-threads: 4

client.event-threads: 4

performance.read-ahead: off

cluster.lookup-optimize: on

performance.client-io-threads: on

performance.cache-size: 1GB

transport.address-family: inet

performance.readdir-ahead: on

nfs.disable: on

cluster.readdir-optimize: on

Thank you.

Sincerely,
Artem

--
Founder, Android Police, APK Mirror, Illogical Robot LLC

beerpla.net | +ArtemRussakovskii | @ArtemR

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users