Hi Nithya, apologies for the long mail... the backend developer told me that he simply uses standard java file utils; test if a directory exists - if not throw exception. A little explanation to our setup: our frontend requests a few files, and via round robin they get distributed to 3 tomcats, and they create the directories and files in the glusterfs volume. It's possible that one of the tomcats creates a directory and writes a file, and then another tomcat wants to write another file but then can't - and throws such a exception. After comparing the timestamps in tomcat and in the filesystem there are 2 cases: a directory already exists a couple of days or it was freshly created. Case 1: directory was freshly created. Timestamps say that the directory and one file were created at 07:05:26, and at 07:05:27 there's an error message in tomcat: 2019-02-20 07:05:27,399 WARN Path /data/repository/shared/public/staticmap/271/362/2713628 is existed but it is not directory [[fetch data pool thread 6@xxxxxxxxxxxxxxxxxxx.FileUtils:747]] java.nio.file.FileAlreadyExistsException: /data/repository/shared/public/staticmap/271/362/2713628 ls -lah --full-time /data/repository/shared/public/staticmap/271/362/2713628/ total 22K drwxr-xr-x 2 tomcat8 tomcat8 4.0K 2019-02-20 07:05:26.420007137 +0100 . drwxr-xr-x 7 tomcat8 tomcat8 4.0K 2019-02-20 07:05:26.411096237 +0100 .. -rw-r--r-- 1 tomcat8 tomcat8 14K 2019-02-20 07:05:26.424007285 +0100 170x100_ed82ac740fe8b9bbaac9ca77aa47f573.jpeg Case 2: directory is already a couple of days old. I just saw an error message in one of the tomcats i immediately checked the 3 paths on the 3 servers: server1: ls /data/repository/shared/public/staticmap/369/217/36921711/ works server2: ls /data/repository/shared/public/staticmap/369/217/36921711/ works server3: ls /data/repository/shared/public/staticmap/369/217/36921711/ ls: cannot access '/data/repository/shared/public/staticmap/369/217/36921711/': No such file or directory Error message in tomcat: 2019-02-20 08:49:28,762 WARN Path /data/repository/shared/public/staticmap/369/217/36921711 is existed but it is not directory [[fetch data pool thread 43@xxxxxxxxxxxxxxxxxxx.FileUtils:747]] java.nio.file.FileAlreadyExistsException: /data/repository/shared/public/staticmap/369/217/36921711 Timestamps of directory (is already ~7 days old): ls -lah --full-time /data/repository/shared/public/staticmap/369/217/36921711/ total 197K drwxr-xr-x 2 tomcat8 tomcat8 4.0K 2019-02-13 12:02:58.164004209 +0100 . drwxr-xr-x 6 tomcat8 tomcat8 4.0K 2019-02-13 13:36:41.159642935 +0100 .. -rw-r--r-- 1 tomcat8 tomcat8 123K 2019-02-13 12:02:58.172004504 +0100 404x475_d473f1d60fc6842dddc7a5460672722b.jpeg -rw-r--r-- 1 tomcat8 tomcat8 66K 2019-02-13 11:58:50.218874380 +0100 420x237_1a7913cc18f4ec142ebc7c578856fb66.jpeg In both cases the directory itself is OK on the other 2 servers, there's an error message on only one of them, including the 'ls $dir -> no such file or directory' error. Interestingly, in both cases, the error gets fixed when either doing umount/mount of glusterfs volume or doing this on the server with the error message: ls /data/repository/shared/public/staticmap/369/217/36921711 -> TAB -> wait 1-2 seconds -> last / appears -> Enter -> directory shows up. First i thought that might be some caching problem, but that seems not quite probable with a directory that's 7 days old. Regards, Hubert Am Mi., 20. Feb. 2019 um 06:12 Uhr schrieb Nithya Balachandran <nbalacha@xxxxxxxxxx>: > > > > On Tue, 19 Feb 2019 at 15:18, Hu Bert <revirii@xxxxxxxxxxxxxx> wrote: >> >> Hello @ll, >> >> one of our backend developers told me that, in the tomcat logs, he >> sees errors that directories on a glusterfs mount aren't readable. >> Within tomcat the errors look like this: >> >> 2019-02-19 07:39:27,124 WARN Path >> /data/repository/shared/public/staticmap/370/626 is existed but it is >> not directory >> java.nio.file.FileAlreadyExistsException: >> /data/repository/shared/public/staticmap/370/626 > > > Do you know which operation failed here? > regards, > Nithya >> >> >> But the basic directory does exist, has been created on 2019-02-18 >> (and is readable on other clients): >> >> ls -lah /data/repository/shared/public/staticmap/370/626/ >> total 36K >> drwxr-xr-x 9 tomcat8 tomcat8 4.0K Feb 18 12:15 . >> drwxr-xr-x 522 tomcat8 tomcat8 4.0K Feb 19 10:29 .. >> drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 11:45 37062632 >> drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 19 09:29 37062647 >> drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 12:15 37062663 >> drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 11:18 37062668 >> drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 11:36 37062681 >> drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 16:53 37062682 >> drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 19 08:19 37062688 >> >> gluster v5.3, debian stretch. >> gluster volume info: https://pastebin.com/UBVWSUex >> gluster volume status: https://pastebin.com/3guxFq5m >> >> mount options on client: >> gluster1:/workdata /data/repository/shared/public glusterfs >> defaults,_netdev,lru-limit=0,backup-volfile-servers=gluster2:gluster3 >> 0 0 >> >> brick mount options: >> /dev/md/4 /gluster/md4 xfs inode64,noatime,nodiratime 0 0 >> >> Hmm... problem with mount options? Or is some cache involved? >> >> >> Best regards, >> Hubert >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> https://lists.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
