Hi, Raghavendra, Thanks a lots for your update! We will test it upon your instructions, and will update you when has results. Thanks & Best Regards, George -----Original Message----- From: Raghavendra Gowdappa [mailto:rgowdapp@xxxxxxxxxx] Sent: Thursday, October 13, 2016 8:58 PM To: Lian, George (Nokia - CN/Hangzhou) <george.lian@xxxxxxxxx> Cc: Gluster-devel@xxxxxxxxxxx; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS <I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@xxxxxxxxxxxxxxxx>; Zhang, Bingxuan (Nokia - CN/Hangzhou) <bingxuan.zhang@xxxxxxxxx>; Zizka, Jan (Nokia - CZ/Prague) <jan.zizka@xxxxxxxxx> Subject: Re: Issue about the size of fstat is less than the really size of the syslog file ----- Original Message ----- > From: "George Lian (Nokia - CN/Hangzhou)" <george.lian@xxxxxxxxx> > To: Gluster-devel@xxxxxxxxxxx > Cc: "I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS" <I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX@xxxxxxxxxxxxxxxx>, "Bingxuan Zhang (Nokia > - CN/Hangzhou)" <bingxuan.zhang@xxxxxxxxx>, "Jan Zizka (Nokia - CZ/Prague)" <jan.zizka@xxxxxxxxx> > Sent: Thursday, October 13, 2016 2:33:53 PM > Subject: Issue about the size of fstat is less than the really size of the syslog file > > Hi, Dear Expert, > We have use glusterfs as a network filesystem, and syslog store in there, > some clients on different host may write the syslog file via “glusterfs” > mount point. > Now we encounter an issue when we “tail” the syslog file, it will occasional > failed with error “ file truncated ” > As we study and trace with the “tail” source code, it failed with the > following code: > if ( S_ISREG (mode) && stats.st_size < f[i].size ) > { > error (0, 0, _("%s: file truncated"), quotef (name)); > /* Assume the file was truncated to 0, > and therefore output all "new" data. */ > xlseek (fd, 0, SEEK_SET, name); > f[i].size = 0; > } > When stats.st_size < f[i].size, what mean the size report by fstat is less > than “tail” had read, it lead to “file truncated”, we also use “strace” > tools to trace the tail application, the related tail strace log as the > below: > nanosleep({1, 0}, NULL) = 0 > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0 > nanosleep({1, 0}, NULL) = 0 > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0 > nanosleep({1, 0}, NULL) = 0 > fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0 > nanosleep({1, 0}, NULL) = 0 > fstat(3, {st_mode=S_IFREG|0644, st_size=192544549, ...}) = 0 > read(3, " Data … -"..., 8192) = 1444 > read(3, " Data.. "..., 8192) = 720 > read(3, "", 8192) = 0 > fstat(3, {st_mode=S_IFREG|0644, st_size=192544789, ...}) = 0 > write(1, “DATA…..” ) = 2164 > write(2, "tail: ", 6tail: ) = 6 > write(2, "/mnt/log/master/syslog: file tru"..., 38/mnt/log/master/syslog: > file truncated) = 38 > as the above strace log, tail has read 1444+720=2164 bytes, > but fstat tell “tail” 192544789 – 192543105 = 1664 which less than 2164, so > it lead to “tail” application “file truncated”. > And if we turn off “write-behind” feature, the issue will not be reproduced > any more. That seems strange. There are no writes happening on the fd/inode through which tail is reading/stating from. So, it seems strange that write-behind is involved here. I suspect whether any of md-cache/read-ahead/io-cache is causing the issue. Can you, 1. Turn off md-cache, read-ahead, io-cache xlators 2. mount glusterfs with --attribute-timeout=0 3. set write-behind on and rerun the tests? If you don't hit the issue, you can experiment by turning on/off of md-cache, read-ahead and io-cache translators and see what are the minimal number of xlators that need to be turned off to not hit the issue (with write-behind on)? regards, Raghavendra > So we think it may be related to cache consistence issue due to performance > consider, but we still have concern that: > The syslog file is used only with “Append” mode, so the size of file > shouldn’t be reduced, when a client read the file, why “fstat” can’t return > the really size match to the cache? > From current investigation, we doubt that the current implement of > “glusterfs” has a bug on “fstat” when cache is on. > Your comments is our highly appreciated! > Thanks & Best Regards > George > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxxx > http://www.gluster.org/mailman/listinfo/gluster-devel _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-devel
