On Thu, May 5, 2016 at 4:59 PM, Serkan Çoban <cobanserkan@xxxxxxxxx> wrote: > Ah I see how can I overlook this, my bad sorry everyone for taking > time to help me... >>BTW, how large is the volume you have? > 9PB usable :) Happy to help! > > Serkan > > On Thu, May 5, 2016 at 2:07 PM, Xavier Hernandez <xhernandez@xxxxxxxxxx> wrote: >> On 05/05/16 11:31, Kaushal M wrote: >>> >>> On Thu, May 5, 2016 at 2:36 PM, David Gossage >>> <dgossage@xxxxxxxxxxxxxxxxxx> wrote: >>>> >>>> >>>> >>>> >>>> On Thu, May 5, 2016 at 3:28 AM, Serkan Çoban <cobanserkan@xxxxxxxxx> >>>> wrote: >>>>> >>>>> >>>>> Hi, >>>>> >>>>> You can find the output below link: >>>>> https://www.dropbox.com/s/wzrh5yp494ogksc/status_detail.txt?dl=0 >>>>> >>>>> Thanks, >>>>> Serkan >>>> >>>> >>>> >>>> Maybe not issue, but playing one of these things is not like the other I >>>> notice of all the bricks only one seems to be different at a quick glance >>>> >>>> Brick : Brick 1.1.1.235:/bricks/20 >>>> TCP Port : 49170 >>>> RDMA Port : 0 >>>> Online : Y >>>> Pid : 26736 >>>> File System : ext4 >>>> Device : /dev/mapper/vol0-vol_root >>>> Mount Options : rw,relatime,data=ordered >>>> Inode Size : 256 >>>> Disk Space Free : 86.1GB >>>> Total Disk Space : 96.0GB >>>> Inode Count : 6406144 >>>> Free Inodes : 6381374 >>>> >>>> Every other brick seems to be 7TB and xfs but this one. >>> >>> >>> Looks like the brick fs isn't mounted, and the root-fs is being used >>> instead. But that still leaves enough inodes free. >>> >>> What I suspect is that one of the cluster translators is mixing up >>> stats when aggregating from multiple bricks. >>> From the log snippet you gave in the first mail, it seems like the >>> disperse translator is possibly involved. >> >> >> Currently ec takes the number of potential files in the subvolume (f_files) >> as the maximum of all its subvolumes, but it takes the available count >> (f_ffree) as the minumum of all its volumes. >> >> This causes max to be ~781.000.000, but free will be ~6.300.000. This gives >> a ~0.8% available, i.e. almost 100% full. >> >> Given the circumstances I think it's the correct thing to do. >> >> Xavi >> >> >>> >>> BTW, how large is the volume you have? Those are a lot of bricks! >>> >>> ~kaushal >>> >>> >>>> >>>> >>>> >>>>> >>>>> >>>>> On Thu, May 5, 2016 at 9:33 AM, Xavier Hernandez <xhernandez@xxxxxxxxxx> >>>>> wrote: >>>>>> >>>>>> Can you post the result of 'gluster volume status v0 detail' ? >>>>>> >>>>>> >>>>>> On 05/05/16 06:49, Serkan Çoban wrote: >>>>>>> >>>>>>> >>>>>>> Hi, Can anyone suggest something for this issue? df, du has no issue >>>>>>> for the bricks yet one subvolume not being used by gluster.. >>>>>>> >>>>>>> On Wed, May 4, 2016 at 4:40 PM, Serkan Çoban <cobanserkan@xxxxxxxxx> >>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I changed cluster.min-free-inodes to "0". Remount the volume on >>>>>>>> clients. inode full messages not coming to syslog anymore but I see >>>>>>>> disperse-56 subvolume still not being used. >>>>>>>> Anything I can do to resolve this issue? Maybe I can destroy and >>>>>>>> recreate the volume but I am not sure It will fix this issue... >>>>>>>> Maybe the disperse size 16+4 is too big should I change it to 8+2? >>>>>>>> >>>>>>>> On Tue, May 3, 2016 at 2:36 PM, Serkan Çoban <cobanserkan@xxxxxxxxx> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> I also checked the df output all 20 bricks are same like below: >>>>>>>>> /dev/sdu1 7.3T 34M 7.3T 1% /bricks/20 >>>>>>>>> >>>>>>>>> On Tue, May 3, 2016 at 1:40 PM, Raghavendra G >>>>>>>>> <raghavendra@xxxxxxxxxxx> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, May 2, 2016 at 11:41 AM, Serkan Çoban >>>>>>>>>> <cobanserkan@xxxxxxxxx> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 1. What is the out put of du -hs <back-end-export>? Please get >>>>>>>>>>>> this >>>>>>>>>>>> information for each of the brick that are part of disperse. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sorry. I needed df output of the filesystem containing brick. Not >>>>>>>>>> du. >>>>>>>>>> Sorry >>>>>>>>>> about that. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> There are 20 bricks in disperse-56 and the du -hs output is like: >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 1.8M /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> 80K /bricks/20 >>>>>>>>>>> >>>>>>>>>>> I see that gluster is not writing to this disperse set. All other >>>>>>>>>>> disperse sets are filled 13GB but this one is empty. I see >>>>>>>>>>> directory >>>>>>>>>>> structure created but no files in directories. >>>>>>>>>>> How can I fix the issue? I will try to rebalance but I don't think >>>>>>>>>>> it >>>>>>>>>>> will write to this disperse set... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G >>>>>>>>>>> <raghavendra@xxxxxxxxxxx> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban >>>>>>>>>>>> <cobanserkan@xxxxxxxxx> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, I cannot get an answer from user list, so asking to devel >>>>>>>>>>>>> list. >>>>>>>>>>>>> >>>>>>>>>>>>> I am getting [dht-diskusage.c:277:dht_is_subvol_filled] >>>>>>>>>>>>> 0-v0-dht: >>>>>>>>>>>>> inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider >>>>>>>>>>>>> adding more bricks. >>>>>>>>>>>>> >>>>>>>>>>>>> message on client logs.My cluster is empty there are only a >>>>>>>>>>>>> couple >>>>>>>>>>>>> of >>>>>>>>>>>>> GB files for testing. Why this message appear in syslog? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> dht uses disk usage information from backend export. >>>>>>>>>>>> >>>>>>>>>>>> 1. What is the out put of du -hs <back-end-export>? Please get >>>>>>>>>>>> this >>>>>>>>>>>> information for each of the brick that are part of disperse. >>>>>>>>>>>> 2. Once you get du information from each brick, the value seen by >>>>>>>>>>>> dht >>>>>>>>>>>> will >>>>>>>>>>>> be based on how cluster/disperse aggregates du info (basically >>>>>>>>>>>> statfs >>>>>>>>>>>> fop). >>>>>>>>>>>> >>>>>>>>>>>> The reason for 100% disk usage may be, >>>>>>>>>>>> In case of 1, backend fs might be shared by data other than >>>>>>>>>>>> brick. >>>>>>>>>>>> In case of 2, some issues with aggregation. >>>>>>>>>>>> >>>>>>>>>>>>> Is is safe to >>>>>>>>>>>>> ignore it? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> dht will try not to have data files on the subvol in question >>>>>>>>>>>> (v0-disperse-56). Hence lookup cost will be two hops for files >>>>>>>>>>>> hashing >>>>>>>>>>>> to >>>>>>>>>>>> disperse-56 (note that other fops like read/write/open still have >>>>>>>>>>>> the >>>>>>>>>>>> cost >>>>>>>>>>>> of single hop and dont suffer from this penalty). Other than that >>>>>>>>>>>> there >>>>>>>>>>>> is >>>>>>>>>>>> no significant harm unless disperse-56 is really running out of >>>>>>>>>>>> space. >>>>>>>>>>>> >>>>>>>>>>>> regards, >>>>>>>>>>>> Raghavendra >>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Gluster-devel mailing list >>>>>>>>>>>>> Gluster-devel@xxxxxxxxxxx >>>>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Raghavendra G >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Gluster-devel mailing list >>>>>>>>>>> Gluster-devel@xxxxxxxxxxx >>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Raghavendra G >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users@xxxxxxxxxxx >>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users@xxxxxxxxxxx >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users@xxxxxxxxxxx >>>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users@xxxxxxxxxxx >>> http://www.gluster.org/mailman/listinfo/gluster-users >>> >> _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
