On Tue, May 22, 2012 at 3:35 AM, alfred de sollize <desollize at gmail.com>wrote: > Hi All, > We tried as Amar suggested the client io-threads but still I > get following errors in the log files > > > ================================================================================= > [2012-05-21 17:54:56.491110] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-21 17:56:30.328936] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-21 18:02:16.150848] D [server-resolve.c:256:resolve_path_deep] > 0-/home9: RESOLVE RENAME() seeking deep resolution of > /tia/mom4p1/work/noriver/ascii/19610401.fms.out > [2012-05-21 18:02:16.151193] D [server-resolve.c:201:resolve_deep_cbk] > 0-home-export-server: /tia/mom4p1/work/noriver/ascii/19610401.fms.out: > failed to resolve (No such file or directory) > [2012-05-21 18:02:16.151231] D > [server-resolve.c:170:resolve_deep_continue] 0-home-export-server: return > value of resolve_*_simple 1 > [2012-05-21 18:04:16.468] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=4 > [2012-05-21 18:04:16.533] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=3 > [2012-05-21 18:04:16.602] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=2 > [2012-05-21 18:04:31.348] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=1 > [2012-05-21 18:22:12.295387] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-21 18:24:03.353140] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-21 18:24:05.764889] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-21 18:26:27.378861] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-21 19:03:37.89056] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: > /home9/tia/mom4p1/work/noriver/ascii/.19610401.fms.out.swp (Function not > implemented) > [2012-05-21 19:04:23.410529] D [io-threads.c:2017:__iot_workers_scale] > 0-home-export-io-threads: scaled threads to 2 (queue_size=4/2) > [2012-05-21 19:05:09.355671] D [io-threads.c:2017:__iot_workers_scale] > 0-home-export-io-threads: scaled threads to 3 (queue_size=8/3) > [2012-05-21 19:05:18.109547] D [io-threads.c:2017:__iot_workers_scale] > 0-home-export-io-threads: scaled threads to 4 (queue_size=16/4) > [2012-05-21 19:05:38.403356] D [io-threads.c:2017:__iot_workers_scale] > 0-home-export-io-threads: scaled threads to 5 (queue_size=32/5) > [2012-05-21 19:10:37.850018] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: /home9/gfskurt/kurt/CCSM/new/scripts/i128/.hf.swp > (Function not implemented) > [2012-05-21 19:37:29.997392] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: /home9/tia/mom4p1/work/noriver/.ibhosts.swp (Function > not implemented) > [2012-05-21 19:55:02.254952] D [server-resolve.c:256:resolve_path_deep] > 0-/home9: RESOLVE OPEN() seeking deep resolution of > /tia/mom4p1/work/noriver/diag_table > [2012-05-21 19:55:02.387242] D [server-resolve.c:256:resolve_path_deep] > 0-/home9: RESOLVE RENAME() seeking deep resolution of > /tia/mom4p1/work/noriver/ascii/19610401.fms.out > [2012-05-21 19:55:02.387592] D [server-resolve.c:201:resolve_deep_cbk] > 0-home-export-server: /tia/mom4p1/work/noriver/ascii/19610401.fms.out: > failed to resolve (No such file or directory) > [2012-05-21 19:55:02.387624] D > [server-resolve.c:170:resolve_deep_continue] 0-home-export-server: return > value of resolve_*_simple 1 > [2012-05-21 19:55:26.576525] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: > /home9/tia/mom4p1/work/noriver/ascii/.19610401.fms.out.swp (Function not > implemented) > [2012-05-21 20:00:01.264940] D [server-resolve.c:256:resolve_path_deep] > 0-/home9: RESOLVE UNLINK() seeking deep resolution of > /tia/mom4p1/work/noriver/history/19610401.ice_diag.nc > [2012-05-21 20:00:01.306673] D [server-resolve.c:256:resolve_path_deep] > 0-/home9: RESOLVE UNLINK() seeking deep resolution of > /tia/mom4p1/work/noriver/history/19610401.ocean_diag.nc > [2012-05-21 20:00:01.969034] D [server-resolve.c:256:resolve_path_deep] > 0-/home9: RESOLVE UNLINK() seeking deep resolution of > /tia/mom4p1/work/noriver/history/19610401.ocean_temp.nc > [2012-05-21 20:01:14.414520] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: > /home9/tia/mom4p1/work/noriver/ascii/.19610401.fms.out.swp (Function not > implemented) > [2012-05-21 20:08:20.756821] D [server-resolve.c:256:resolve_path_deep] > 0-/home9: RESOLVE RENAME() seeking deep resolution of > /tia/mom4p1/work/noriver/ascii/no_time_stamp.logfile.000000.out > [2012-05-21 20:08:20.757176] D [server-resolve.c:201:resolve_deep_cbk] > 0-home-export-server: > /tia/mom4p1/work/noriver/ascii/no_time_stamp.logfile.000000.out: failed to > resolve (No such file or directory) > [2012-05-21 20:08:20.757214] D > [server-resolve.c:170:resolve_deep_continue] 0-home-export-server: return > value of resolve_*_simple 1 > [2012-05-21 20:17:50.736328] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: /home9/tia/mom4p1/work/noriver/.ibhosts.swp (Function > not implemented) > [2012-05-21 20:26:34.648108] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: /home9/tia/mom4p1/work/noriver/.ibhosts.swp (Function > not implemented) > [2012-05-21 20:33:56.293882] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: > /home9/gfskurt/kurt/CCSM/new/scratch/i512/ocn/input/gx1v6_tavg_contents > (Function not implemented) > [2012-05-21 20:34:14.798261] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: > /home9/gfskurt/kurt/CCSM/new/scripts/i512/CaseDocs/ice_in (Function not > implemented) > [2012-05-21 20:34:14.840494] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: > /home9/gfskurt/kurt/CCSM/new/scripts/i512/CaseDocs/ice_in (Function not > implemented) > [2012-05-21 20:34:14.933004] D [posix.c:769:posix_do_chmod] > 0-home-export-posix: > /home9/gfskurt/kurt/CCSM/new/scripts/i512/CaseDocs/ice_in (Function not > implemented) > [2012-05-21 20:36:42.411] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=4 > [2012-05-21 20:36:42.533] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=3 > [2012-05-21 20:36:42.571] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=2 > [2012-05-21 20:39:29.233] D [io-threads.c:118:iot_worker] > 0-home-export-io-threads: timeout, terminated. conf->curr_count=1 > [2012-05-21 22:24:07.309485] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-21 22:24:07.466879] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-22 02:48:07.317096] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > [2012-05-22 02:48:09.155900] D [posix.c:316:posix_lstat_with_gfid] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/features/access-control.so(posix_acl_readdirp+0x208) > [0x7f16165edda8] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_readdirp+0xf) > [0x7f16168025df] > (-->/opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/storage/posix.so(posix_do_readdir+0x655) > [0x7f1616802505]))) 0-home-export-posix: failed to get gfid > > > ============================================================================================================= > after which this brick would get disconnected and we get files missing and > TENC errors > Is there a limitation on simultaneous i-o that could be handled by > glusterfs? > reg > Al > > > On Tue, May 15, 2012 at 11:24 AM, Amar Tumballi <amarts at redhat.com> wrote: > >> On 05/15/2012 09:02 AM, alfred de sollize wrote: >> >>> Has anybody worked with ccsm , wrf on Glusterfs? These spawn huge number >>> of threads for IO. >>> What are xlators to disable and what oher tune-ups so it stops giving >>> TENC >>> Al >>> >>> >>> >> Considering you have lot of threads doing I/O, can you try 'gluster >> volume set <VOLNAME> client-io-threads enable' and see if it makes any >> difference? >> >> Regards, >> Amar >> >> >>> >>> >>> On Sun, May 13, 2012 at 10:54 PM, alfred de sollize <desollize at gmail.com >>> <mailto:desollize at gmail.com>> wrote: >>> >>> Hi Amar, >>> The version is in subject line - is >>> Glusterfs-3.2.5 >>> >>> TENC comes in clients for commands like - [cd /home] or [df -h >>> /home] >>> /home - Transport End point not connected >>> it also appears in server logs, >>> Attaching some part of client and server logs for one of the error >>> bricks >>> is some problem for application integrations also? We use Centos-6.1.. >>> Al >>> >>> >>> >>> >>> >>> >>> >>> On Fri, May 11, 2012 at 11:32 AM, Amar Tumballi <amarts at redhat.com >>> <mailto:amarts at redhat.com>> wrote: >>> >>> On 05/10/2012 09:39 PM, alfred de sollize wrote: >>> >>> We are setting up a 180 node cluster for weather modeling. 2 >>> Storage >>> servers with 32GB Ram each. QDR INfiniband interconnect. >>> When we run iozone with 1GB perthread (128Kb blocksize) from >>> 32 clients >>> (2 iozone threads per client). >>> The run succeeds however run fails for 64 clients and we >>> start getting >>> "Transport Endpoint not connected" errors. >>> >>> There are 10 bricks(5 from each server ) each of ~4.2TB >>> making 42TB of >>> export-volume that is fuse-mounted on the clients. >>> >>> There is no other error in the log files except for "TENC" . >>> >>> >>> When you say log file, which files are you looking at. Also, >>> which version of the glusterfs are you running ? >>> >>> -Amar >>> >>> >>> >>> >>> >>> ______________________________**_________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/**mailman/listinfo/gluster-users<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20120522/3dda8a66/attachment-0001.htm>
