Re: Problem with massive file renaming in glusterfs volume

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Jose,

On Sat, Sep 4, 2021 at 1:58 PM José Manuel Blanco <josemanuel.blanco@xxxxxx> wrote:

Hi all,

We have a problem with tasks (written in PHP) doing a lot of file renaming/moving (even several renames per second).

The pattern is always the same: the task rename files with variable filename to the SAME final file (that is: the destination filename is ALWAYS the same).

Problem: according to glusterfs and task logs, some rename are successful but other don't and we don't understand why or what is causing the error in the failing renames because the error is "File exists", but the rename supossedly must address these situations correctly and "overwrite" the destination file if it exists (it uses the PHP rename() function)

More info:

- We use a distributed-replicated volume in a 3 node cluster

gluster volume info moodle-cv
 
Volume Name: moodle-cv
Type: Distributed-Replicate
Volume ID: 1eef9714-3943-4d77-b42f-6b1144389c56
Status: Started
Snapshot Count: 0
Number of Bricks: 5 x (2 + 1) = 15
Transport-type: tcp
Bricks:
Brick1: moodle2017-n2:/glusterfs/moodle-cv/brick1/datos
Brick2: moodle2017-n3:/glusterfs/moodle-cv/brick1/replica
Brick3: moodle2017-n1:/glusterfs/moodle-cv/brick1/arbiter (arbiter)
Brick4: moodle2017-n3:/glusterfs/moodle-cv/brick2/datos
Brick5: moodle2017-n2:/glusterfs/moodle-cv/brick2/replica
Brick6: moodle2017-n1:/glusterfs/moodle-cv/brick2/arbiter (arbiter)
Brick7: moodle2017-n2:/glusterfs/moodle-cv/brick3/datos
Brick8: moodle2017-n3:/glusterfs/moodle-cv/brick3/replica
Brick9: moodle2017-n1:/glusterfs/moodle-cv/brick3/arbiter (arbiter)
Brick10: moodle2017-n3:/glusterfs/moodle-cv/brick4/datos
Brick11: moodle2017-n2:/glusterfs/moodle-cv/brick4/replica
Brick12: moodle2017-n1:/glusterfs/moodle-cv/brick4/arbiter (arbiter)
Brick13: moodle2017-n2:/glusterfs/moodle-cv/brick5/datos
Brick14: moodle2017-n3:/glusterfs/moodle-cv/brick5/replica
Brick15: moodle2017-n1:/glusterfs/moodle-cv/brick5/arbiter (arbiter)
Options Reconfigured:
nfs.disable: on
storage.fips-mode-rchecksum: on
performance.open-behind: off
performance.lazy-open: no
cluster.self-heal-daemon: enable

- The nodes use Oracle Linux 7.9 (RedHat clone) and GlusterFS 8.5

- The tasks use the PHP rename() function

- The tasks renaming files are executed IN ONE NODE OF THE CLUSTER that mounts the volume using the FUSE client

- At the end of the message I paste a (very) little fragment of the volume log


Any ideas of the possible cause of the problem and/or suggestions to avoid it?


Most probably the issue is caused by special internal files needed by Gluster that are used to reference the correct location of a file in a distributed volume (they are referred as linkto files). I would say the error happens because those files already exist when it's expected that they don't exist (I think it's a similar problem as https://github.com/gluster/glusterfs/issues/1723).

Are you using FUSE mounts ? are you doing the renames from more than one client ?

Regards,

Xavi

Regards.


This is the fragment of the log:

* I've "separated" the lines by "rename operation" but all the lines appear together in the log

--->SUCCESSFUL RENAME:

[2021-09-03 09:04:02.478321] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e50272f326.07618090.temp (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (11fdf019-16f8-48ae-bd1f-05b670b29ec1) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-2)

--->UNSUCCESSFUL RENAME: 

[2021-09-03 09:04:02.497845] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp (350698df-35d9-4489-90cc-faa004bfc861) (hash=moodle-cv-replicate-0/cache=moodle-cv-replicate-0) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

[2021-09-03 09:04:02.503271] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-0: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.503339] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-1: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.503393] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-2: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.504997] W [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse: 781438412: /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502771a44.52737408.temp -> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache => -1 (File exists)

--->UNSUCCESSFUL RENAME:

[2021-09-03 09:04:02.729314] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp (5bff6629-80b8-4d9c-8e37-abfa7691124e) (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

[2021-09-03 09:04:02.735631] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.735709] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.735747] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.737356] W [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse: 781438861: /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502b00578.34807541.temp -> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache => -1 (File exists)

--->SUCCESSFUL RENAME:

[2021-09-03 09:04:02.799776] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502c175a4.68869416.temp (82725bab-b888-4d24-b239-31b86f50425f) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (4a875eac-5389-400c-9359-458e151054f7) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

--->UNSUCCESFULL RENAME:

[2021-09-03 09:04:02.963545] I [MSGID: 109066] [dht-rename.c:1955:dht_rename] 2-moodle-cv-dht: renaming /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp (b3c5435c-f79b-402f-ba00-0e78dcdb6cd2) (hash=moodle-cv-replicate-1/cache=moodle-cv-replicate-1) => /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache (82725bab-b888-4d24-b239-31b86f50425f) (hash=moodle-cv-replicate-3/cache=moodle-cv-replicate-3) 

[2021-09-03 09:04:02.968665] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-5: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.968801] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-4: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.968847] W [MSGID: 114031] [client-rpc-fops_v2.c:2464:client4_0_link_cbk] 2-moodle-cv-client-3: remote operation failed. [{source=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp}, {target=/2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache}, {errno=17}, {error=File exists}]

[2021-09-03 09:04:02.970438] W [fuse-bridge.c:2500:fuse_rename_cbk] 0-glusterfs-fuse: 781439391: /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/Cb511LuXTg.6131e502e973d1.55328482.temp -> /2021_2022/cache/cachestore_file/default_application/core_eventinvalidation/5ea-cache/5ea912b69c8e3655eb5275f8c49b7a7265066e4d.cache => -1 (File exists)


--

Jose Manuel Blanco

Analista Sistemas · josemanuel.blanco@xxxxxx
Tfno.: (+34) 983 18 6411

Universidad de Valladolid
Servicio de las TIC

Edificio Alfonso VIII · C/ Real de Burgos s/n
47011 · Valladolid · España (Spain)

| Facebook | Twitter | UVa

Este mensaje puede contener información confidencial, sometida al secreto profesional, cuya divulgación no está permitida por la ley. Si usted no es su destinatario, por favor, notifíquelo al remitente y borre este correo de su sistema. A los efectos de la protección de datos y el RGPD, consulte: protección de datos en la UVa. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posibles perjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. Piensa en verde: lee en la pantalla.

This message may contain confidential information covered by the obligation of professional secrecy, the disclosure of which would be contrary to the law. If you are not the intended recipient, please advise the sender and delete this e-mail from your system. For the purposes of data protection and GDPR, read: protección de datos en la UVa. The sender does not guarantee the integrity, the accuracy, the swift delivery or the security of this email transmission, and assumes no responsibility for any possible damage incurred through data capture, virus incorporation or any manipulation carried out by third parties. Think green: read on the screen.

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux