Re: Issue with duplicated files in gluster 3.10

Luca Gervasi <luca.gervasi@xxxxxxxxx> · Mon, 13 Mar 2017 09:38:56 +0000

Hello Ravishankar,we had to change the directory as we fixed that one, so please check these links which refers to a new (broken) path:

https://nopaste.me/view/1ee13a63 LS debug log
https://nopaste.me/view/80ac1e13 getfattr
https://nopaste.me/view/eafb0b44 volume status

Thanks in advance.

Luca & Andrea

On Sat, 11 Mar 2017 at 02:01 Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

    On 03/10/2017 10:32 PM, Luca Gervasi
      wrote:

      Hi,
        I'm Andrea's collegue. I'd like to add that we have no
          trusted.afr xattr on the root folder

    Just to confirm, this would be 'includes2013' right?

         where those files are located and every file seems to be
          clean on each brick.
        You can find another example file's xattr here: https://nopaste.me/view/3c2014ac
        Here a listing: https://nopaste.me/view/eb4430a2
        This behavior causes the directory which contains those
          files undeletable (we had to clear them up on brick level,
          clearing all the hardlinks too).
        This issue is visible on fuse mounted volumes while it's
          not noticeable when mounted in NFS through ganesha.

    Could you provide the complete output of `gluster volume info`? I
    want to find out which bricks constitute a replica pair.

    Also could you change the diagnostics.client-log-level to DEBUG
    temporarily, do an `ls <directory where you see duplicate
    entries>` on the fuse mount  and share the corresponding mount
    log?

    Thanks,

    Ravi

        Thanks a lot.

        Luca Gervasi

        On Fri, 10 Mar 2017 at 17:41 Andrea Fogazzi <foggy@xxxxxxxxxxx>
          wrote:

                Hi community,
                we ran an extensive issue on
                  our installation of gluster 3.10, which we did
                  upgraded from 3.8.8 (it's a distribute+replicate, 5
                  nodes, 3 bricks in replica 2+1 quorum); recently we
                  noticed a frequent issue where files get duplicated on
                  the some of the directories; this is visible on the
                  fuse mount points (RW), but not on the NFS/Ganesha
                  (RO) mount points.

                A sample of an ll output:

                ---------T 1 48 web_rw     0 Mar
                  10 11:57 paginazione.shtml
                -rw-rw-r-- 1 48 web_rw   272 Feb
                  18 22:00 paginazione.shtml

                As you can see, the file is listed twice, but only one
                of the two is good (the name is identical, we verified
                that no spurious/hidden characters are present in the
                name); the issue maybe is related on how we uploaded the
                files on the file system, via incremental rsync on the
                fuse mount.

                Do anyone have suggestion on how it can happen, how to
                solve existing duplication or how to prevent to happen
                anymore.

                Thanks in advance.

                Best regards,

                andrea

                Options Reconfigured:
                performance.cache-invalidation:
                  true
                cluster.favorite-child-policy:
                  mtime
                features.cache-invalidation: 1
                network.inode-lru-limit: 90000
                performance.cache-size: 1024MB
                storage.linux-aio: on
                nfs.outstanding-rpc-limit: 64
                storage.build-pgfid: on
                cluster.server-quorum-type:
                  server
                cluster.self-heal-daemon: enable
                performance.nfs.io-cache: on
                performance.client-io-threads: on
                performance.nfs.stat-prefetch: on
                performance.nfs.io-threads: on
                diagnostics.latency-measurement:
                  on
                diagnostics.count-fop-hits: on
                performance.md-cache-timeout: 1
                performance.io-thread-count: 16
                performance.high-prio-threads: 32
                performance.normal-prio-threads:
                  32
                performance.low-prio-threads: 32
                performance.least-prio-threads: 1
                nfs.acl: off
                nfs.rpc-auth-unix: off
                diagnostics.client-log-level:
                  ERROR
                diagnostics.brick-log-level:
                  ERROR
                cluster.lookup-unhashed: auto
                performance.nfs.quick-read: on
                performance.nfs.read-ahead: on
                cluster.quorum-type: auto
                cluster.locking-scheme: granular
                cluster.data-self-heal-algorithm:
                  full
                transport.address-family: inet
                performance.readdir-ahead: on
                nfs.disable: on
                cluster.lookup-optimize: on
                cluster.readdir-optimize: on
                performance.read-ahead: off
                performance.write-behind-window-size:
                  1MB
                client.event-threads: 4
                server.event-threads: 16
                cluster.granular-entry-heal:
                  enable
                performance.parallel-readdir: on
                cluster.server-quorum-ratio: 51

                Andrea Fogazzi

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users