Re: Problem with massive file renaming in glusterfs volume

José Manuel Blanco <josemanuel.blanco@xxxxxx> · Mon, 13 Sep 2021 14:50:37 +0200

    Hello Xavier,
    Sorry: these days I've
          been busy with other tasks :-(

    More info about the
          problem and your questions:
    - The rename/move errors
          appear not only in the glusterfs node that execute tasks
          renaming files BUT ALSO in ALL clients of the glusterfs volume
    - All clients (and even
          the glusterfs node executing the failing renames) mount the
          glusterfs volume using the fuse client
    Regards.

              Jose Manuel Blanco

                Analista Sistemas · josemanuel.blanco@xxxxxx

                Tfno.: (+34) 983 18 6411

              Universidad de Valladolid

                Servicio de las TIC

                Edificio Alfonso VIII · C/ Real de Burgos s/n

                  47011 · Valladolid · España (Spain)
              | Facebook | Twitter
                | UVa 

              Este mensaje puede contener información confidencial,
                sometida al secreto profesional, cuya divulgación no
                está permitida por la ley. Si usted no es su
                destinatario, por favor, notifíquelo al remitente y
                borre este correo de su sistema. A los efectos de la
                protección de datos y el RGPD, consulte: protección de datos en la UVa. El
                emisor no garantiza la integridad, rapidez o seguridad
                del presente correo, ni se responsabiliza de posibles
                perjuicios derivados de la captura, incorporaciones de
                virus o cualesquiera otras manipulaciones efectuadas por
                terceros. Piensa en verde: lee en
                  la pantalla.
              This message may contain confidential information
                covered by the obligation of professional secrecy, the
                disclosure of which would be contrary to the law. If you
                are not the intended recipient, please advise the sender
                and delete this e-mail from your system. For the
                purposes of data protection and GDPR, read: protección de datos en la UVa. The
                sender does not guarantee the integrity, the accuracy,
                the swift delivery or the security of this email
                transmission, and assumes no responsibility for any
                possible damage incurred through data capture, virus
                incorporation or any manipulation carried out by third
                parties. Think green: read on the
                  screen.

    El 6/9/21 a las 9:40, Xavi Hernandez
      escribió:

        Hi Jose,

          On Sat, Sep 4, 2021 at 1:58
            PM José Manuel Blanco <josemanuel.blanco@xxxxxx>
            wrote:

              Hi all,
              We have a
                    problem with tasks (written in PHP) doing a lot of
                    file renaming/moving (even several renames per
                    second).
              The pattern is
                    always the same: the task rename files with variable
                    filename to the SAME final file (that is: the
                    destination filename is ALWAYS the same).
              Problem:
                    according to glusterfs and task logs, some rename
                    are successful but other don't and we don't
                    understand why or what is causing the error in the
                    failing renames because the error is "File exists",
                    but the rename supossedly must address these
                    situations correctly and "overwrite" the destination
                    file if it exists (it uses the PHP rename()
                    function)

              More info:

              - We use a
                    distributed-replicated volume in a 3 node cluster

                gluster volume
                      info moodle-cv

                    Volume Name:
                    moodle-cv

                  Type: Distributed-Replicate

                  Volume ID:
                    1eef9714-3943-4d77-b42f-6b1144389c56

                  Status: Started

                  Snapshot Count: 0

                  Number of Bricks: 5 x (2 + 1) =
                    15

                  Transport-type: tcp

                  Bricks:

                  Brick1:
                    moodle2017-n2:/glusterfs/moodle-cv/brick1/datos

                  Brick2:
                    moodle2017-n3:/glusterfs/moodle-cv/brick1/replica

                  Brick3:
                    moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter
                    (arbiter)

                  Brick4:
                    moodle2017-n3:/glusterfs/moodle-cv/brick2/datos

                  Brick5:
                    moodle2017-n2:/glusterfs/moodle-cv/brick2/replica

                  Brick6:
                    moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter
                    (arbiter)

                  Brick7:
                    moodle2017-n2:/glusterfs/moodle-cv/brick3/datos

                  Brick8:
                    moodle2017-n3:/glusterfs/moodle-cv/brick3/replica

                  Brick9:
                    moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter
                    (arbiter)

                  Brick10:
                    moodle2017-n3:/glusterfs/moodle-cv/brick4/datos

                  Brick11:
                    moodle2017-n2:/glusterfs/moodle-cv/brick4/replica

                  Brick12:
                    moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter
                    (arbiter)

                  Brick13:
                    moodle2017-n2:/glusterfs/moodle-cv/brick5/datos

                  Brick14:
                    moodle2017-n3:/glusterfs/moodle-cv/brick5/replica

                  Brick15:
                    moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter
                    (arbiter)

                  Options Reconfigured:

                  nfs.disable: on

                  storage.fips-mode-rchecksum: on

                  performance.open-behind: off

                  performance.lazy-open: no

                  cluster.self-heal-daemon:
                    enable

              - The nodes use
                    Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5
              - The tasks use
                    the PHP rename() function

              - The tasks
                    renaming files are executed IN ONE NODE OF THE
                    CLUSTER that mounts the volume using the FUSE client
              - At the end of
                    the message I paste a (very) little fragment of the
                    volume log

              Any ideas of the
                    possible cause of the problem and/or suggestions to
                    avoid it?

          Most probably the issue is caused by special internal
            files needed by Gluster that are used to reference the
            correct location of a file in a distributed volume (they are
            referred as linkto files). I would say the error happens
            because those files already exist when it's expected that
            they don't exist (I think it's a similar problem as https://github.com/gluster/glusterfs/issues/1723).

          Are you using FUSE mounts ? are you doing the renames
            from more than one client ?

          Regards,

          Xavi

              Regards.

              This is the
                    fragment of the log:
              * I've
                    "separated" the lines by "rename operation" but all
                    the lines appear together in the log

                --->SUCCESSFUL RENAME:
                [2021-09-03 09:04:02.478321] I
                    [MSGID: 109066] [dht-rename.c:1955:dht_rename]
                    2-moodle-cv-dht: renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp
                    (4a875eac-5389-400c-9359-458e151054f7)
                    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
                    =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    (11fdf019-16f8-48ae-bd1f-05b670b29ec1)
                    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)
                --->UNSUCCESSFUL RENAME:  

                [2021-09-03 09:04:02.497845] I
                    [MSGID: 109066] [dht-rename.c:1955:dht_rename]
                    2-moodle-cv-dht: renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
                    (350698df-35d9-4489-90cc-faa004bfc861)
                    (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0)
                    =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    (4a875eac-5389-400c-9359-458e151054f7)
                    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

                [2021-09-03 09:04:02.503271] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-0: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.503339] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-1: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.503393] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-2: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.504997] W
                    [fuse-bridge.c:2500:fuse_rename_cbk]
                    0-glusterfs-fuse: 781438412:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
                    ->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    => -1 (File exists)
                --->UNSUCCESSFUL RENAME:

                [2021-09-03 09:04:02.729314] I
                    [MSGID: 109066] [dht-rename.c:1955:dht_rename]
                    2-moodle-cv-dht: renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
                    (5bff6629-80b8-4d9c-8e37-abfa7691124e)
                    (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1)
                    =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    (4a875eac-5389-400c-9359-458e151054f7)
                    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

                [2021-09-03 09:04:02.735631] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-5: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.735709] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-4: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.735747] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-3: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.737356] W
                    [fuse-bridge.c:2500:fuse_rename_cbk]
                    0-glusterfs-fuse: 781438861:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
                    ->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    => -1 (File exists)

                --->SUCCESSFUL RENAME:
                [2021-09-03 09:04:02.799776] I
                    [MSGID: 109066] [dht-rename.c:1955:dht_rename]
                    2-moodle-cv-dht: renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp
                    (82725bab-b888-4d24-b239-31b86f50425f)
                    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)
                    =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    (4a875eac-5389-400c-9359-458e151054f7)
                    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

                --->UNSUCCESFULL RENAME:

                [2021-09-03 09:04:02.963545] I
                    [MSGID: 109066] [dht-rename.c:1955:dht_rename]
                    2-moodle-cv-dht: renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
                    (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2)
                    (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1)
                    =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    (82725bab-b888-4d24-b239-31b86f50425f)
                    (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

                [2021-09-03 09:04:02.968665] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-5: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.968801] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-4: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.968847] W
                    [MSGID: 114031]
                    [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
                    2-moodle-cv-client-3: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
                    {errno=17}, {error=File exists}] 

                [2021-09-03 09:04:02.970438] W
                    [fuse-bridge.c:2500:fuse_rename_cbk]
                    0-glusterfs-fuse: 781439391:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
                    ->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
                    => -1 (File exists)

              -- 

                        Jose Manuel Blanco

                          Analista Sistemas · josemanuel.blanco@xxxxxx

                          Tfno.: (+34) 983 18 6411

                        Universidad de Valladolid

                          Servicio de las TIC

                          Edificio Alfonso VIII · C/ Real de
                            Burgos s/n

                            47011 · Valladolid · España (Spain)
                        | Facebook
                          | Twitter
                          | UVa 

                        Este mensaje puede contener información
                          confidencial, sometida al secreto profesional,
                          cuya divulgación no está permitida por la ley.
                          Si usted no es su destinatario, por favor,
                          notifíquelo al remitente y borre este correo
                          de su sistema. A los efectos de la protección
                          de datos y el RGPD, consulte: protección
                            de datos en la UVa. El emisor no
                          garantiza la integridad, rapidez o seguridad
                          del presente correo, ni se responsabiliza de
                          posibles perjuicios derivados de la captura,
                          incorporaciones de virus o cualesquiera otras
                          manipulaciones efectuadas por terceros. Piensa
                            en verde: lee en la pantalla.
                        This message may contain confidential
                          information covered by the obligation of
                          professional secrecy, the disclosure of which
                          would be contrary to the law. If you are not
                          the intended recipient, please advise the
                          sender and delete this e-mail from your
                          system. For the purposes of data protection
                          and GDPR, read: protección
                            de datos en la UVa. The sender does not
                          guarantee the integrity, the accuracy, the
                          swift delivery or the security of this email
                          transmission, and assumes no responsibility
                          for any possible damage incurred through data
                          capture, virus incorporation or any
                          manipulation carried out by third parties. Think
                            green: read on the screen.

            ________

            Community Meeting Calendar:

            Schedule -

            Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC

            Bridge: https://meet.google.com/cpu-eiue-hvk

            Gluster-users mailing list

            Gluster-users@xxxxxxxxxxx

            https://lists.gluster.org/mailman/listinfo/gluster-users

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users