Re: Problem with massive file renaming in glusterfs volume

Xavi Hernandez <jahernan@xxxxxxxxxx> · Mon, 6 Sep 2021 09:40:06 +0200

Hi Jose,
On Sat, Sep 4, 2021 at 1:58 PM José Manuel Blanco <josemanuel.blanco@xxxxxx> wrote:

    Hi all,
    We have a problem with
          tasks (written in PHP) doing a lot of file renaming/moving
          (even several renames per second).
    The pattern is always the
          same: the task rename files with variable filename to the SAME
          final file (that is: the destination filename is ALWAYS the
          same).
    Problem: according to
          glusterfs and task logs, some rename are successful but other
          don't and we don't understand why or what is causing the error
          in the failing renames because the error is "File exists", but
          the rename supossedly must address these situations correctly
          and "overwrite" the destination file if it exists (it uses the
          PHP rename() function)

    More info:

    - We use a
          distributed-replicated volume in a 3 node cluster

      gluster volume info
            moodle-cv

          Volume Name: moodle-cv

        Type: Distributed-Replicate

        Volume ID:
          1eef9714-3943-4d77-b42f-6b1144389c56

        Status: Started

        Snapshot Count: 0

        Number of Bricks: 5 x (2 + 1) = 15

        Transport-type: tcp

        Bricks:

        Brick1:
          moodle2017-n2:/glusterfs/moodle-cv/brick1/datos

        Brick2:
          moodle2017-n3:/glusterfs/moodle-cv/brick1/replica

        Brick3:
          moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter)

        Brick4:
          moodle2017-n3:/glusterfs/moodle-cv/brick2/datos

        Brick5:
          moodle2017-n2:/glusterfs/moodle-cv/brick2/replica

        Brick6:
          moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter)

        Brick7:
          moodle2017-n2:/glusterfs/moodle-cv/brick3/datos

        Brick8:
          moodle2017-n3:/glusterfs/moodle-cv/brick3/replica

        Brick9:
          moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter)

        Brick10:
          moodle2017-n3:/glusterfs/moodle-cv/brick4/datos

        Brick11:
          moodle2017-n2:/glusterfs/moodle-cv/brick4/replica

        Brick12:
          moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter)

        Brick13:
          moodle2017-n2:/glusterfs/moodle-cv/brick5/datos

        Brick14:
          moodle2017-n3:/glusterfs/moodle-cv/brick5/replica

        Brick15:
          moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter)

        Options Reconfigured:

        nfs.disable: on

        storage.fips-mode-rchecksum: on

        performance.open-behind: off

        performance.lazy-open: no

        cluster.self-heal-daemon: enable

    - The nodes use Oracle
          Linux 7.9 (RedHat clone) and GlusterFS 8.5
    - The tasks use the PHP
          rename() function

    - The tasks renaming files
          are executed IN ONE NODE OF THE CLUSTER that mounts the volume
          using the FUSE client
    - At the end of the
          message I paste a (very) little fragment of the volume log

    Any ideas of the possible
          cause of the problem and/or suggestions to avoid it?

Most probably the issue is caused by special internal files needed by Gluster that are used to reference the correct location of a file in a distributed volume (they are referred as linkto files). I would say the error happens because those files already exist when it's expected that they don't exist (I think it's a similar problem as https://github.com/gluster/glusterfs/issues/1723).

Are you using FUSE mounts ? are you doing the renames from more than one client ?

Regards,

Xavi

    Regards.

    This is the fragment of
          the log:
    * I've "separated" the
          lines by "rename operation" but all the lines appear together
          in the log

      --->SUCCESSFUL RENAME:
      [2021-09-03 09:04:02.478321] I [MSGID:
          109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
          renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp
          (4a875eac-5389-400c-9359-458e151054f7)
          (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          (11fdf019-16f8-48ae-bd1f-05b670b29ec1)
          (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)
      --->UNSUCCESSFUL RENAME:  

      [2021-09-03 09:04:02.497845] I [MSGID:
          109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
          renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
          (350698df-35d9-4489-90cc-faa004bfc861)
          (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          (4a875eac-5389-400c-9359-458e151054f7)
          (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)  

      [2021-09-03 09:04:02.503271] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-0: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.503339] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-1: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.503393] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-2: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.504997] W
          [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
          781438412:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp
          ->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          => -1 (File exists)
      --->UNSUCCESSFUL RENAME:

      [2021-09-03 09:04:02.729314] I [MSGID:
          109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
          renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
          (5bff6629-80b8-4d9c-8e37-abfa7691124e)
          (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          (4a875eac-5389-400c-9359-458e151054f7)
          (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)  

      [2021-09-03 09:04:02.735631] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-5: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.735709] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-4: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.735747] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-3: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.737356] W
          [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
          781438861:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp
          ->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          => -1 (File exists)

      --->SUCCESSFUL RENAME:
      [2021-09-03 09:04:02.799776] I [MSGID:
          109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
          renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp
          (82725bab-b888-4d24-b239-31b86f50425f)
          (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          (4a875eac-5389-400c-9359-458e151054f7)
          (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)  

      --->UNSUCCESFULL RENAME:

      [2021-09-03 09:04:02.963545] I [MSGID:
          109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht:
          renaming
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
          (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2)
          (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) =>
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          (82725bab-b888-4d24-b239-31b86f50425f)
          (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3)  

      [2021-09-03 09:04:02.968665] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-5: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.968801] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-4: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.968847] W [MSGID:
          114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk]
          2-moodle-cv-client-3: remote operation failed.
[{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp},
{target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache},
          {errno=17}, {error=File exists}] 

      [2021-09-03 09:04:02.970438] W
          [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse:
          781439391:
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp
          ->
/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache
          => -1 (File exists)

    -- 

              Jose Manuel Blanco

                Analista Sistemas · josemanuel.blanco@xxxxxx

                Tfno.: (+34) 983 18 6411

              Universidad de Valladolid

                Servicio de las TIC

                Edificio Alfonso VIII · C/ Real de Burgos s/n

                  47011 · Valladolid · España (Spain)
              | Facebook | Twitter
                | UVa 

              Este mensaje puede contener información confidencial,
                sometida al secreto profesional, cuya divulgación no
                está permitida por la ley. Si usted no es su
                destinatario, por favor, notifíquelo al remitente y
                borre este correo de su sistema. A los efectos de la
                protección de datos y el RGPD, consulte: protección de datos en la UVa. El
                emisor no garantiza la integridad, rapidez o seguridad
                del presente correo, ni se responsabiliza de posibles
                perjuicios derivados de la captura, incorporaciones de
                virus o cualesquiera otras manipulaciones efectuadas por
                terceros. Piensa en verde: lee en
                  la pantalla.
              This message may contain confidential information
                covered by the obligation of professional secrecy, the
                disclosure of which would be contrary to the law. If you
                are not the intended recipient, please advise the sender
                and delete this e-mail from your system. For the
                purposes of data protection and GDPR, read: protección de datos en la UVa. The
                sender does not guarantee the integrity, the accuracy,
                the swift delivery or the security of this email
                transmission, and assumes no responsibility for any
                possible damage incurred through data capture, virus
                incorporation or any manipulation carried out by third
                parties. Think green: read on the
                  screen.

________

Community Meeting Calendar:

Schedule -

Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC

Bridge: https://meet.google.com/cpu-eiue-hvk

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

https://lists.gluster.org/mailman/listinfo/gluster-users

________

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users