No, that conclusion was too early. Error happens on all clients, but the directories are different, e.g.: dir1 fails on client1, but works on all others dir2 fails on client2, but works on all others etc. Interesting: when doing a 'ls' on the dir and TAB->autocomplete (missing the last number), the directory appears: ls /data/repository/shared/public/images/370/435/3704359 37043592/ 37043593/ 37043595/ 37043597/ 37043598/ 37043599/ and the can be opened and cd'ed into. Content appears: ls -lah /shared/public/images/370/435/37043597/ total 1.6M drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 19 13:52 . drwxr-xr-x 69 tomcat8 tomcat8 4.0K Feb 17 21:42 .. -rw-r--r-- 1 tomcat8 tomcat8 7.0K Feb 19 13:52 100x100f.jpg -rw-r--r-- 1 tomcat8 tomcat8 235K Feb 19 13:23 1080x610r.jpg -rw-r--r-- 1 tomcat8 tomcat8 96K Feb 18 10:46 1200x1200s.jpg -rw-r--r-- 1 tomcat8 tomcat8 14K Feb 17 23:03 150x150f.jpg [...] Some error message in /var/log/glusterfs/data-repository-shared-public.log on one of the clients saying that mkdir fails because the directory exists: [2019-02-19 10:39:44.673279] W [fuse-bridge.c:582:fuse_entry_cbk] 0-glusterfs-fuse: 69757484: MKDIR() /images/370/435/37043597 => -1 (File exists) [2019-02-19 11:14:58.625014] W [fuse-bridge.c:582:fuse_entry_cbk] 0-glusterfs-fuse: 69977210: MKDIR() /images/370/435/37043597 => -1 (File exists) [2019-02-19 11:42:15.626959] W [fuse-bridge.c:582:fuse_entry_cbk] 0-glusterfs-fuse: 70148412: MKDIR() /images/370/435/37043597 => -1 (File exists) [2019-02-19 12:26:01.065483] W [fuse-bridge.c:582:fuse_entry_cbk] 0-glusterfs-fuse: 70406931: MKDIR() /images/370/435/37043597 => -1 (File exists) The directory exists -> warning is OK, but why doesn't it appear first? Am Di., 19. Feb. 2019 um 14:10 Uhr schrieb Hu Bert <revirii@xxxxxxxxxxxxxx>: > > a little update: it seems to be only one client. Whenever there's a > "no such file or directory" error in the logs, the directory can be > read/opened on all the other clients. very strange... > > Nothing in glusterfs logs besides the zillion "dict is NULL [Invalid > argument]" warnings. Must be something on the client itself. > > Am Di., 19. Feb. 2019 um 10:47 Uhr schrieb Hu Bert <revirii@xxxxxxxxxxxxxx>: > > > > Hello @ll, > > > > one of our backend developers told me that, in the tomcat logs, he > > sees errors that directories on a glusterfs mount aren't readable. > > Within tomcat the errors look like this: > > > > 2019-02-19 07:39:27,124 WARN Path > > /data/repository/shared/public/staticmap/370/626 is existed but it is > > not directory > > java.nio.file.FileAlreadyExistsException: > > /data/repository/shared/public/staticmap/370/626 > > > > But the basic directory does exist, has been created on 2019-02-18 > > (and is readable on other clients): > > > > ls -lah /data/repository/shared/public/staticmap/370/626/ > > total 36K > > drwxr-xr-x 9 tomcat8 tomcat8 4.0K Feb 18 12:15 . > > drwxr-xr-x 522 tomcat8 tomcat8 4.0K Feb 19 10:29 .. > > drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 11:45 37062632 > > drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 19 09:29 37062647 > > drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 12:15 37062663 > > drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 11:18 37062668 > > drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 11:36 37062681 > > drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 18 16:53 37062682 > > drwxr-xr-x 2 tomcat8 tomcat8 4.0K Feb 19 08:19 37062688 > > > > gluster v5.3, debian stretch. > > gluster volume info: https://pastebin.com/UBVWSUex > > gluster volume status: https://pastebin.com/3guxFq5m > > > > mount options on client: > > gluster1:/workdata /data/repository/shared/public glusterfs > > defaults,_netdev,lru-limit=0,backup-volfile-servers=gluster2:gluster3 > > 0 0 > > > > brick mount options: > > /dev/md/4 /gluster/md4 xfs inode64,noatime,nodiratime 0 0 > > > > Hmm... problem with mount options? Or is some cache involved? > > > > > > Best regards, > > Hubert _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
