Re: Erroneous "No space left on device." messages

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




HI,

After some more poking around in the logs (specifically the brick logs)

Note brick1 & brick2 are on one server, brick3 and brick4 are on the second server.

Pat


On 3/10/20 11:51 AM, Pat Haley wrote:

Hi,

We have developed a problem with Gluster reporting "No space left on device." even though "df" of both the gluster filesystem and the underlying bricks show space available (details below).  Our inode usage is between 1-3%.  We are running gluster 3.7.11 in a distributed volume across 2 servers (2 bricks each). We have followed the thread https://lists.gluster.org/pipermail/gluster-users/2020-March/037821.html but haven't found a solution yet.

Last night we ran a rebalance which appeared successful (and have since cleared up some more space which seems to have mainly been on one brick).  There were intermittent erroneous "No space..." messages last night, but they have become much more frequent today.

Any help would be greatly appreciated.

Thanks

---------------------------
[root@mseas-data2 ~]# df -h
---------------------------
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb        164T  164T  324G 100% /mnt/brick2
/dev/sda        164T  164T  323G 100% /mnt/brick1
---------------------------
[root@mseas-data2 ~]# df -i
---------------------------
Filesystem         Inodes    IUsed      IFree IUse% Mounted on
/dev/sdb       1375470800 31207165 1344263635    3% /mnt/brick2
/dev/sda       1384781520 28706614 1356074906    3% /mnt/brick1

---------------------------
[root@mseas-data3 ~]# df -h
---------------------------
/dev/sda               91T   91T  323G 100% /export/sda/brick3
/dev/mapper/vg_Data4-lv_Data4
                       91T   88T  3.4T  97% /export/sdc/brick4
---------------------------
[root@mseas-data3 ~]# df -i
---------------------------
/dev/sda              679323496  9822199  669501297    2% /export/sda/brick3
/dev/mapper/vg_Data4-lv_Data4
                     3906272768 11467484 3894805284    1% /export/sdc/brick4



---------------------------------------
[root@mseas-data2 ~]# gluster --version
---------------------------------------
glusterfs 3.7.11 built on Apr 27 2016 14:09:22
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.



-----------------------------------------
[root@mseas-data2 ~]# gluster volume info
-----------------------------------------
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Brick3: mseas-data3:/export/sda/brick3
Brick4: mseas-data3:/export/sdc/brick4
Options Reconfigured:
nfs.export-volumes: off
nfs.disable: on
performance.readdir-ahead: on
diagnostics.brick-sys-log-level: WARNING
nfs.exports-auth-enable: on
server.allow-insecure: on
auth.allow: *
disperse.eager-lock: off
performance.open-behind: off
performance.md-cache-timeout: 60
network.inode-lru-limit: 50000
diagnostics.client-log-level: ERROR



--------------------------------------------------------------
[root@mseas-data2 ~]# gluster volume status data-volume detail
--------------------------------------------------------------
Status of volume: data-volume
------------------------------------------------------------------------------
Brick                : Brick mseas-data2:/mnt/brick1
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 4601
File System          : xfs
Device               : /dev/sda
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 318.8GB
Total Disk Space     : 163.7TB
Inode Count          : 1365878288
Free Inodes          : 1337173596
------------------------------------------------------------------------------
Brick                : Brick mseas-data2:/mnt/brick2
TCP Port             : 49155
RDMA Port            : 0
Online               : Y
Pid                  : 7949
File System          : xfs
Device               : /dev/sdb
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 319.8GB
Total Disk Space     : 163.7TB
Inode Count          : 1372421408
Free Inodes          : 1341219039
------------------------------------------------------------------------------
Brick                : Brick mseas-data3:/export/sda/brick3
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 4650
File System          : xfs
Device               : /dev/sda
Mount Options        : rw
Inode Size           : 512
Disk Space Free      : 325.3GB
Total Disk Space     : 91.0TB
Inode Count          : 692001992
Free Inodes          : 682188893
------------------------------------------------------------------------------
Brick                : Brick mseas-data3:/export/sdc/brick4
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 23772
File System          : xfs
Device               : /dev/mapper/vg_Data4-lv_Data4
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 3.4TB
Total Disk Space     : 90.9TB
Inode Count          : 3906272768
Free Inodes          : 3894809903

-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley@xxxxxxx
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301
________



Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux