Continuing gfs2 problems: Am I doing something wrong????

Scooter Morris <scooter@xxxxxxxxxxxx> · Tue, 03 Aug 2010 12:38:17 -0700



    HI all,

        We continue to have gfs2 crashes and hangs on our production
    cluster, so I'm beginning to think that we've done something really
    wrong.  Here is our set-up:

    
      4 node cluster, only 3 participate in gfs2 filesystems
      Running several services on multiple nodes using gfs2:
      
        IMAP (dovecot)
        Web (apache with lots of python)
        Samba (using ctdb)
      
      GFS2 partitions are multipathed on an HP EVA-based SAN (no
        LVM) -- here is fstab from one node (the three nodes are all the
        same):
    
    
      LABEL=/1                /                      
          ext3    defaults        1 1

        LABEL=/boot1            /boot                   ext3   
          defaults        1 2

        tmpfs                   /dev/shm                tmpfs  
          defaults        0 0

        devpts                  /dev/pts                devpts 
          gid=5,mode=620  0 0

        sysfs                   /sys                    sysfs  
          defaults        0 0

        proc                    /proc                   proc   
          defaults        0 0

        LABEL=SW-cciss/c0d0p2   swap                    swap   
          defaults        0 0

        LABEL=plato:Mail        /var/spool/mail         gfs2   
          noatime,_netdev

        LABEL=plato:VarTmp      /var/tmp                gfs2   
          _netdev

        LABEL=plato:UsrLocal    /usr/local              gfs2   
          noatime,_netdev

        LABEL=plato:UsrLocalProjects /usr/local/projects gfs2  
          noatime,_netdev

        LABEL=plato:Home2       /home/socr              gfs2   
          noatime,_netdev

        LABEL=plato:HomeNoBackup /home/socr/nobackup    gfs2   
          _netdev

        LABEL=plato:DbBackup    /databases/backups      gfs2   
          noatime,_netdev

        LABEL=plato:DbMol       /databases/mol          gfs2   
          noatime,_netdev

        LABEL=plato:MolDbBlast  /databases/mol/blast    gfs2   
          noatime,_netdev

        LABEL=plato:MolDbEmboss /databases/mol/emboss   gfs2   
          noatime,_netdev
    
    
      Kernel version is: 2.6.18-194.3.1.el5 and all nodes are
        x86_64.
      What's happening is every so often, we start seeing
        gfs2-related task hangs in the logs.  In the last instance (last
        Friday) we've got this:
    
    
      Node 0:

        [2010-07-30 13:23:25]INFO: task imap:25716
            blocked for more than 120 seconds.^M

          [2010-07-30 13:23:25]"echo 0 >
            /proc/sys/kernel/hung_task_timeout_secs" disables this
            message.^M

          [2010-07-30 13:23:25]imap          D ffff8100010825a0    
            0 25716   9217         24080 25667 (NOTLB)^M

          [2010-07-30 13:23:25] ffff810619b59bc8 0000000000000086
            ffff810113233f10 ffffffff00000000^M

          [2010-07-30 13:23:26] ffff81000f8c5cd0 000000000000000a
            ffff810233416040 ffff81082fd05100^M

          [2010-07-30 13:23:26] 00012196d153c88e 0000000000008b81
            ffff810233416228 0000000f6a949180^M

          [2010-07-30 13:23:26]Call Trace:^M

          [2010-07-30 13:23:26] [<ffffffff887d0be6>]
            :gfs2:gfs2_dirent_find+0x0/0x4e^M

          [2010-07-30 13:23:26] [<ffffffff887d0c18>]
            :gfs2:gfs2_dirent_find+0x32/0x4e^M

          [2010-07-30 13:23:26] [<ffffffff887d5ee7>]
            :gfs2:just_schedule+0x0/0xe^M

          [2010-07-30 13:23:26] [<ffffffff887d5ef0>]
            :gfs2:just_schedule+0x9/0xe^M

          [2010-07-30 13:23:26] [<ffffffff80063a16>]
            __wait_on_bit+0x40/0x6e^M

          [2010-07-30 13:23:26] [<ffffffff887d5ee7>]
            :gfs2:just_schedule+0x0/0xe^M

          [2010-07-30 13:23:26] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M

          [2010-07-30 13:23:26] [<ffffffff800a0aec>]
            wake_bit_function+0x0/0x23^M

          [2010-07-30 13:23:26] [<ffffffff887d5ee2>]
            :gfs2:gfs2_glock_wait+0x2b/0x30^M

          [2010-07-30 13:23:26] [<ffffffff887e579e>]
            :gfs2:gfs2_permission+0x83/0xd5^M

          [2010-07-30 13:23:26] [<ffffffff887e5796>]
            :gfs2:gfs2_permission+0x7b/0xd5^M

          [2010-07-30 13:23:26] [<ffffffff8000ce97>]
            do_lookup+0x65/0x1e6^M

          [2010-07-30 13:23:26] [<ffffffff8000d918>]
            permission+0x81/0xc8^M

          [2010-07-30 13:23:26] [<ffffffff8000997f>]
            __link_path_walk+0x173/0xf42^M

          [2010-07-30 13:23:26] [<ffffffff8000e9e2>]
            link_path_walk+0x42/0xb2^M

          [2010-07-30 13:23:26] [<ffffffff8000ccb2>]
            do_path_lookup+0x275/0x2f1^M

          [2010-07-30 13:23:26] [<ffffffff8001280e>]
            getname+0x15b/0x1c2^M

          [2010-07-30 13:23:27] [<ffffffff80023876>]
            __user_walk_fd+0x37/0x4c^M

          [2010-07-30 13:23:27] [<ffffffff80028846>]
            vfs_stat_fd+0x1b/0x4a^M

          [2010-07-30 13:23:27] [<ffffffff800638b3>]
            schedule_timeout+0x92/0xad^M

          [2010-07-30 13:23:27] [<ffffffff80097dab>]
            process_timeout+0x0/0x5^M

          [2010-07-30 13:23:27] [<ffffffff800f8435>]
            sys_epoll_wait+0x3b8/0x3f9^M

          [2010-07-30 13:23:27] [<ffffffff800235a8>]
            sys_newstat+0x19/0x31^M

          [2010-07-30 13:23:27] [<ffffffff8005d229>]
            tracesys+0x71/0xe0^M

          [2010-07-30 13:23:27] [<ffffffff8005d28d>]
            tracesys+0xd5/0xe0^M

        
        Node 1:

        [2010-07-30 13:23:59]INFO: task pdflush:623
            blocked for more than 120 seconds.^M

          [2010-07-30 13:23:59]"echo 0 >
            /proc/sys/kernel/hung_task_timeout_secs" disables this
            message.^M

          [2010-07-30 13:23:59]pdflush       D ffff810407069aa0    
            0   623    291           624   622 (L-TLB)^M

          [2010-07-30 13:23:59] ffff8106073c1bd0 0000000000000046
            0000000000000001 ffff8103fea899a8^M

          [2010-07-30 13:23:59] ffff8106073c1c30 000000000000000a
            ffff8105fff7c0c0 ffff8107fff4c820^M

          [2010-07-30 13:24:00] 0000ed85d9d7a027 0000000000011b50
            ffff8105fff7c2a8 00000006f0a9d0d0^M

          [2010-07-30 13:24:00]Call Trace:^M

          [2010-07-30 13:24:00] [<ffffffff8001a927>]
            submit_bh+0x10a/0x111^M

          [2010-07-30 13:24:00] [<ffffffff88802ee7>]
            :gfs2:just_schedule+0x0/0xe^M

          [2010-07-30 13:24:00] [<ffffffff88802ef0>]
            :gfs2:just_schedule+0x9/0xe^M

          [2010-07-30 13:24:00] [<ffffffff80063a16>]
            __wait_on_bit+0x40/0x6e^M

          [2010-07-30 13:24:00] [<ffffffff88802ee7>]
            :gfs2:just_schedule+0x0/0xe^M

          [2010-07-30 13:24:00] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M

          [2010-07-30 13:24:00] [<ffffffff800a0aec>]
            wake_bit_function+0x0/0x23^M

          [2010-07-30 13:24:00] [<ffffffff88802ee2>]
            :gfs2:gfs2_glock_wait+0x2b/0x30^M

          [2010-07-30 13:24:00] [<ffffffff88813269>]
            :gfs2:gfs2_write_inode+0x5f/0x152^M

          [2010-07-30 13:24:00] [<ffffffff88813261>]
            :gfs2:gfs2_write_inode+0x57/0x152^M

          [2010-07-30 13:24:00] [<ffffffff8002fbf8>]
            __writeback_single_inode+0x1e9/0x328^M

          [2010-07-30 13:24:00] [<ffffffff80020ec9>]
            sync_sb_inodes+0x1b5/0x26f^M

          [2010-07-30 13:24:00] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M

          [2010-07-30 13:24:00] [<ffffffff8005123a>]
            writeback_inodes+0x82/0xd8^M

          [2010-07-30 13:24:00] [<ffffffff800c97b5>]
            wb_kupdate+0xd4/0x14e^M

          [2010-07-30 13:24:00] [<ffffffff80056879>]
            pdflush+0x0/0x1fb^M

          [2010-07-30 13:24:00] [<ffffffff800569ca>]
            pdflush+0x151/0x1fb^M

          [2010-07-30 13:24:00] [<ffffffff800c96e1>]
            wb_kupdate+0x0/0x14e^M

          [2010-07-30 13:24:01] [<ffffffff80032894>]
            kthread+0xfe/0x132^M

          [2010-07-30 13:24:01] [<ffffffff8009d734>]
            request_module+0x0/0x14d^M

          [2010-07-30 13:24:01] [<ffffffff8005dfb1>]
            child_rip+0xa/0x11^M

          [2010-07-30 13:24:01] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M

          [2010-07-30 13:24:01] [<ffffffff80032796>]
            kthread+0x0/0x132^M

          [2010-07-30 13:24:01] [<ffffffff8005dfa7>]
            child_rip+0x0/0x11^M

        
        Node 2:

        [2010-07-30 13:24:46]INFO: task
            delete_workqueu:7175 blocked for more than 120 seconds.^M

          [2010-07-30 13:24:46]"echo 0 >
            /proc/sys/kernel/hung_task_timeout_secs" disables this
            message.^M

          [2010-07-30 13:24:46]delete_workqu D ffff81082b5cf860    
            0  7175    329          7176  7174 (L-TLB)^M

          [2010-07-30 13:24:46] ffff81081ed6dbf0 0000000000000046
            0000000000000018 ffffffff887a84f3^M

          [2010-07-30 13:24:46] 0000000000000286 000000000000000a
            ffff81082dd477e0 ffff81082b5cf860^M

          [2010-07-30 13:24:46] 00012166bf7ec21d 000000000002ed0b
            ffff81082dd479c8 00000007887a9e5a^M

          [2010-07-30 13:24:46]Call Trace:^M

          [2010-07-30 13:24:46] [<ffffffff887a84f3>]
            :dlm:request_lock+0x93/0xa0^M

          [2010-07-30 13:24:47] [<ffffffff8884f556>]
            :lock_dlm:gdlm_ast+0x0/0x311^M

          [2010-07-30 13:24:47] [<ffffffff8884f2c1>]
            :lock_dlm:gdlm_bast+0x0/0x8d^M

          [2010-07-30 13:24:47] [<ffffffff887d3ee7>]
            :gfs2:just_schedule+0x0/0xe^M

          [2010-07-30 13:24:47] [<ffffffff887d3ef0>]
            :gfs2:just_schedule+0x9/0xe^M

          [2010-07-30 13:24:47] [<ffffffff80063a16>]
            __wait_on_bit+0x40/0x6e^M

          [2010-07-30 13:24:47] [<ffffffff887d3ee7>]
            :gfs2:just_schedule+0x0/0xe^M

          [2010-07-30 13:24:47] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M

          [2010-07-30 13:24:47] [<ffffffff800a0aec>]
            wake_bit_function+0x0/0x23^M

          [2010-07-30 13:24:47] [<ffffffff887d3ee2>]
            :gfs2:gfs2_glock_wait+0x2b/0x30^M

          [2010-07-30 13:24:47] [<ffffffff887e82cf>]
            :gfs2:gfs2_check_blk_type+0xd7/0x1c9^M

          [2010-07-30 13:24:47] [<ffffffff887e82c7>]
            :gfs2:gfs2_check_blk_type+0xcf/0x1c9^M

          [2010-07-30 13:24:47] [<ffffffff80063ab0>]
            out_of_line_wait_on_bit+0x6c/0x78^M

          [2010-07-30 13:24:47] [<ffffffff887e804f>]
            :gfs2:gfs2_rindex_hold+0x32/0x12b^M

          [2010-07-30 13:24:47] [<ffffffff887d5a29>]
            :gfs2:delete_work_func+0x0/0x65^M

          [2010-07-30 13:24:47] [<ffffffff887d5a29>]
            :gfs2:delete_work_func+0x0/0x65^M

          [2010-07-30 13:24:47] [<ffffffff887e3e3a>]
            :gfs2:gfs2_delete_inode+0x76/0x1b4^M

          [2010-07-30 13:24:47] [<ffffffff887e3e01>]
            :gfs2:gfs2_delete_inode+0x3d/0x1b4^M

          [2010-07-30 13:24:47] [<ffffffff8000d3ba>]
            dput+0x2c/0x114^M

          [2010-07-30 13:24:48] [<ffffffff887e3dc4>]
            :gfs2:gfs2_delete_inode+0x0/0x1b4^M

          [2010-07-30 13:24:48] [<ffffffff8002f35e>]
            generic_delete_inode+0xc6/0x143^M

          [2010-07-30 13:24:48] [<ffffffff887d5a83>]
            :gfs2:delete_work_func+0x5a/0x65^M

          [2010-07-30 13:24:48] [<ffffffff8004d8f0>]
            run_workqueue+0x94/0xe4^M

          [2010-07-30 13:24:48] [<ffffffff8004a12b>]
            worker_thread+0x0/0x122^M

          [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M

          [2010-07-30 13:24:48] [<ffffffff8004a21b>]
            worker_thread+0xf0/0x122^M

          [2010-07-30 13:24:48] [<ffffffff8008d087>]
            default_wake_function+0x0/0xe^M

          [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M

          [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M

          [2010-07-30 13:24:48] [<ffffffff80032894>]
            kthread+0xfe/0x132^M

          [2010-07-30 13:24:48] [<ffffffff8005dfb1>]
            child_rip+0xa/0x11^M

          [2010-07-30 13:24:48] [<ffffffff800a08a6>]
            keventd_create_kthread+0x0/0xc4^M

          [2010-07-30 13:24:48] [<ffffffff80032796>]
            kthread+0x0/0x132^M

          [2010-07-30 13:24:48] [<ffffffff8005dfa7>]
            child_rip+0x0/0x11^M

        
      Various messages related to hung_task_timeouts repeated on
        each node (usually related to imap).
      Within a minute or two, the cluster was completely hung.  Root
        could log into the console, but commands (like dmesg) would just
        hang.
    
    So, my major question:  is there something wrong with my
    configuration?  Have we done something really stupid?  The initial
    response from RedHat was that we shouldn't run services on multiple
    nodes that access gfs2, which seems a little confusing since we
    would use ext3 or ext4 if we were going to node lock (or failover)
    the partitions.  Have we missed something somewhere?

    
    Thanks in advance for any help anyone can give.  We're getting
    pretty desperate here since the downtime is starting to have a
    significant impact on our credibility.

    
    -- scooter

    
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster