Hello,
I have faced strange problem on one of the Gluster client/server nodes. File exist but cannot be deleted from the fuse mount point on one client node, but can be deleted from another
Attemt to delete:
root at SERVER:~# rm /var/lib/nova/instances/instance-0098a323/disk
rm: cannot remove `/var/lib/nova/instances/instance-0098a323/disk': No such file or directory
Although files listed as existing in the volume:
root@ SERVER:~# ls -al /var/lib/nova/instances/instance-0098a323
total 6143442
drwxrwxr-x 2 nova nova 143 Jun 11 11:54 .
drwxr-xr-x 107 nova nova 409600 Jun 11 11:54 ..
-rw-r--r-- 1 root root 6290472960 Jun 11 11:54 disk
root@ SERVER:~# stat /var/lib/nova/instances/instance-0098a323/disk
File: `/var/lib/nova/instances/instance-0098a323/disk'
Size: 6290472960 Blocks: 12286080 IO Block: 131072 regular file
Device: 1eh/30d Inode: 11831107392822693115 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2013-06-03 14:12:03.803403051 +0100
Modify: 2013-06-11 11:54:09.623013613 +0100
Change: 2013-06-11 11:54:12.355063759 +0100
Birth: -
File locartion information:
root at SERVER:~/openstack# getfattr -n trusted.glusterfs.pathinfo /var/lib/nova/instances/instance-0098a323/disk
getfattr: Removing leading '/' from absolute path names
# file: var/lib/nova/instances/instance-0098a323/disk
trusted.glusterfs.pathinfo="(<DISTRIBUTE:glustervmstore-dht> (<REPLICATE:glustervmstore-replicate-21> <POSIX(/mnt/vmstore/brick):BRICK_SERV1:/mnt/vmstore/brick/instance-0098a323/disk> <POSIX(/mnt/vmstore/brick): BRICK_SERV2:/mnt/vmstore/brick/instance-0098a323/disk>))"
Subvolume information:
volume glustervmstore-replicate-3
type cluster/replicate
subvolumes glustervmstore-client-6 glustervmstore-client-7
end-volume
volume glustervmstore-replicate-21
type cluster/replicate
subvolumes glustervmstore-client-42 glustervmstore-client-43
end-volume
It is also available on the replica bricks. However I WAS ABLE to delete the file from another node, which also mounts the same volume under: /var/lib/nova/instances
Log file excerpt (I presume that?s /var/log/gluster/var-lib-nova-instances.log). What is strange that glustervmstore-client-6 and glustervmstore-client-6 does not represent actual brick servers where file is stored:
[2013-06-11 14:10:02.640244] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
[2013-06-11 14:10:51.769641] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:10:51.769737] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:10:51.769769] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204475960: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:18:15.350431] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:18:15.350513] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:18:15.350547] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204480095: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:26:07.418892] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:26:07.418980] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:26:07.419011] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204484135: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:27:12.510324] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
[2013-06-11 14:27:44.518849] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:27:44.518937] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:27:44.518967] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204485011: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:29:33.289316] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
[2013-06-11 14:38:13.487311] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-41: remote operation failed: No such
file or directory
[2013-06-11 14:38:13.487563] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-40: remote operation failed: No such file or directory
The same happened for /instance-0098a327/disk file although there are no similar line
[2013-06-11 14:27:12.510324] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
I was able to delete the file from another node.
There are no mention of the folder in the GlusterFS logs on the actual brick servers.
Thank you
______________________________________________________________________________
Elvinas Piliponis I UNIX Engineer I GTIS UNIX Engineering
Tel +370 5 251 1218, 7 2249 1218 I Mobile +370 656 69249 I Email elvinas.piliponis at barclays.com<mailto:elvinas.piliponis at barclays.com>
Barclays, GreenHall 9th floor 09.E4.1, Up?s g. 21, Vilnius, Lithuania LT-081218
barclays.com
This e-mail and any attachments are confidential and intended
solely for the addressee and may also be privileged or exempt from
disclosure under applicable law. If you are not the addressee, or
have received this e-mail in error, please notify the sender
immediately, delete it from your system and do not copy, disclose
or otherwise act upon any part of this e-mail or its attachments.
Internet communications are not guaranteed to be secure or
virus-free.
The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any
Internet communications by any third party, or from the
transmission of any viruses. Replies to this e-mail may be
monitored by the Barclays Group for operational or business
reasons.
Any opinion or other information in this e-mail or its attachments
that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.
Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.
Barclays Bank PLC is authorised by the Prudential Regulation
Authority and regulated by the Financial Conduct Authority and the
Prudential Regulation Authority (Financial Services Register No.
122702).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130611/6fd9dbb5/attachment-0001.html>
