On Thu, Jun 16, 2022 at 22:59, Pat Haley<phaley@xxxxxxx> wrote:
Hi Strahil,
I poked around our logs, and found this on the front-end (from the day & time of the last time we had the issue)
Jun 15 10:51:17 mseas gdata[155485]: [2022-06-15 14:51:17.263858] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-data-volume-client-2: server 172.16.1.113:49153 has not responded in the last 42 seconds, disconnecting.
This would indicate that the problem is related. For us, however, I believe we can reproduce this issue at will (i.e. simply try to gunzip the same file). Unfortunately I have to go to a meeting now, but if you have some specific tests you'd like me to try, I can try them when I get back.
Thanks
Pat
On 6/16/22 3:07 PM, Strahil Nikolov wrote:
Pat,
Can you check the cpu and disk performance when the volume reports the issue?
It seems that similar issue was reported in https://lists.gluster.org/pipermail/gluster-users/2019-March/035944.html but I don't see a clear solution.Take a look in the thread and check if it matches your symptoms.
Best Regards,Strahil Nikolov
On Thu, Jun 16, 2022 at 18:14, Pat Haley<phaley@xxxxxxx> wrote:
Hi Strahil,
I poked around again and for brick 3 (where the file we were testing resides) I only found the same log file as was at the bottom of my first Email:
---------------------------------------------------
mseas-data3: bricks/export-sda-brick3.log
-----------------------------------------
[2022-06-15 14:50:42.588143] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-data-volume-server: disconnecting connection from mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
[2022-06-15 14:50:42.588220] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on /projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC
[2022-06-15 14:50:42.588259] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on /projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz
[2022-06-15 14:50:42.588288] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-data-volume-server: Shutting down connection mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
[2022-06-15 14:50:53.605215] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-data-volume-server: accepted client from mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-29 (version: 3.7.11)
[2022-06-15 14:50:42.588247] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on /projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC
Thanks
Pat
On 6/15/22 6:47 PM, Strahil Nikolov wrote:
I agree. It will be very hard to debug.
Anything in the brick logs ?
I think it's pointless to mention that EL6 is dead and Gluster v3 is so old that it's worth considering a migration to a newer setup.
Best Regards,Strahil Nikolov
On Wed, Jun 15, 2022 at 22:51, Yaniv Kaul<ykaul@xxxxxxxxxx> wrote:________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley@xxxxxxx Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley@xxxxxxx Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
