Ok. Spec files are not the case. I have: node1 and node2 in afr1 node3 and node4 in afr2 afr1 and afr2 in unify with round robin scheduler. When I have to copy let's say 50GB of small files to /mnt/glusterfs and if file goes to afr1 it may place this file into /node1export/data, /node2export/data and somehow to the /node3export/data (or to afr2 simply). So I won't be able to open this file from /mnt/glusterfs because it exist in each afr and that's not acceptable for glusterfs's unify translator. It should be very clear now. It happened before. when I was using 1.3.7 version (Amar discovered it). But we were unable to hit it again. And now I hit it badly using 1.3.9. It's somehow random bug.. It may cause and may not and it causes only during writing many files at the same time so far. Logs.: 2008-06-28 18:10:59 E [unify.c:335:unify_lookup] unify-test: returning ESTALE for /data/issues/issue_43070/front_94x125.jpg: file count is 3 2008-06-28 18:10:59 E [unify.c:339:unify_lookup] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-test-ns 2008-06-28 18:10:59 E [unify.c:339:unify_lookup] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-data-01 2008-06-28 18:10:59 E [unify.c:339:unify_lookup] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-web-01 2008-06-28 18:10:59 E [fuse-bridge.c:459:fuse_entry_cbk] glusterfs-fuse: 12276369: (34) /data/issues/issue_43070/front_94x125.jpg => -1 (116) 2008-06-28 18:10:59 E [unify.c:873:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: entry_count is 3 2008-06-28 18:10:59 E [unify.c:876:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-web-01 2008-06-28 18:10:59 E [unify.c:876:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-test-ns 2008-06-28 18:10:59 E [unify.c:876:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-data-01 2008-06-28 18:10:59 E [fuse-bridge.c:692:fuse_fd_cbk] glusterfs-fuse: 12276370: (12) /data/issues/issue_43070/front_94x125.jpg => -1 (5) 2008-06-28 18:10:59 E [unify.c:873:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: entry_count is 3 2008-06-28 18:10:59 E [unify.c:876:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-web-01 2008-06-28 18:10:59 E [unify.c:876:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-test-ns 2008-06-28 18:10:59 E [unify.c:876:unify_open] unify-test: /data/issues/issue_43070/front_94x125.jpg: found on afr-data-01 2008-06-28 18:10:59 E [fuse-bridge.c:692:fuse_fd_cbk] glusterfs-fuse: 12276372: (12) /data/issues/issue_43070/front_94x125.jpg => -1 (5) 2008-06-28 18:17:14 E [unify.c:873:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: entry_count is 3 2008-06-28 18:17:14 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: found on afr-data-01 2008-06-28 18:17:14 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: found on afr-test-ns 2008-06-28 18:17:14 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: found on afr-web-01 2008-06-28 18:17:14 E [fuse-bridge.c:692:fuse_fd_cbk] glusterfs-fuse: 12276445: (12) /data/issues/issue_1000004/front_94x125.jpg => -1 (5) 2008-06-28 18:17:14 E [unify.c:873:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: entry_count is 3 2008-06-28 18:17:14 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: found on afr-data-01 2008-06-28 18:17:14 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: found on afr-test-ns 2008-06-28 18:17:14 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000004/front_94x125.jpg: found on afr-web-01 2008-06-28 18:17:14 E [fuse-bridge.c:692:fuse_fd_cbk] glusterfs-fuse: 12276447: (12) /data/issues/issue_1000004/front_94x125.jpg => -1 (5) 2008-06-28 18:17:16 E [unify.c:873:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: entry_count is 3 2008-06-28 18:17:16 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-web-01 2008-06-28 18:17:16 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-data-01 2008-06-28 18:17:16 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-test-ns 2008-06-28 18:17:16 E [fuse-bridge.c:692:fuse_fd_cbk] glusterfs-fuse: 12276483: (12) /data/issues/issue_1000003/front_94x125.jpg => -1 (5) 2008-06-28 18:17:16 E [unify.c:873:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: entry_count is 3 2008-06-28 18:17:16 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-web-01 2008-06-28 18:17:16 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-data-01 2008-06-28 18:17:16 E [unify.c:876:unify_open] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-test-ns 2008-06-28 18:17:16 E [fuse-bridge.c:692:fuse_fd_cbk] glusterfs-fuse: 12276485: (12) /data/issues/issue_1000003/front_94x125.jpg => -1 (5) 2008-06-28 18:17:41 E [unify.c:335:unify_lookup] unify-test: returning ESTALE for /data/issues/issue_1000003/front_94x125.jpg: file count is 3 2008-06-28 18:17:41 E [unify.c:339:unify_lookup] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-web-01 2008-06-28 18:17:41 E [unify.c:339:unify_lookup] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-data-01 2008-06-28 18:17:41 E [unify.c:339:unify_lookup] unify-test: /data/issues/issue_1000003/front_94x125.jpg: found on afr-test-ns And so on. P.S.: my name is Dmitriy :-) -- Best regards, Dmitriy Kotkin (aka diem @ irc)
