On 03-Jun-2017 3:27 AM, "Diego Remolina" <dijuremo@xxxxxxxxx> wrote:
Hi everyone,
Is there anything else we could do to check on this problem and try to
fix it? The issue is definitively related to either the samba vfs
gluster plugin or gluster itself. I am not sure how to pin it down
futher.
I don't think it is vfs plugin because you haven't updated samba packages and vfs plugin is same as before.
It is either gfapi in Gluster or one of the lower xlators.
Changing to fuse based shares might make the problem go away and can be used as a workaround.
I went ahead and created a new share in the samba server which is on a
local filesystem where the OS is installed, not part of gluster:
# mount | grep home
]# ls -ld /home
drwxr-xr-x. 14 root root 4096 Jun 2 17:21 /home
This shows this is not mounted from anywhere. so I created a folder
and shared it:
]# ls -ld /home/rvtsharetest/
drwxrwx---. 3 dijuremo Staff 95 Jun 2 17:31 /home/rvtsharetest/
]# tail -7 /etc/samba/smb.conf
[rvtsharetest]
path=/home/rvtsharetest
browseable = yes
write list = @Staff,dijuremo
guest ok = yes
create mask = 664
directory mask = 775
When accessing Revit files in this new share the problem is *not* observed.
When accessing Revit files on any of the samba shares that use the vfs
gluster plugin and are stored in the gluster volume, we see the
problems.
Further analysis of the issue is even more disconcerting. As you may
remember, I have found a workaround about renaming the file and back
to the original, where things work. Here is where it gets more
interesting. This seems to be workstation dependent, not user
dependent.
1. User1 logs into PC1 and tries to access the file and gets the error
message from Revit "ACCESS DENIED".
2. User1 uses windows explorer to go to the file location (we tried
doing this from the server itself on the command line and it did not
change anything, i.e. su - User1 then mv command). User1 renames the
file, Revit.rvt -> Revit2.rvt, and clicks away for the rename to take
place. User1 immediately renames the file to the original, Revit2.rvt
-> Revit.rvt
3. User1 opens the file and everything works properly.
4. User2 logs into PC1 and tries to open the file. The file opens and
works properly.
5. User2 logs into PC2 and tries to open the file, the problem comes
up. User2 uses the rename trick and this fixes the problem on PC2.
Even if User1 now comes to use PC2, User1 will have no problems with
the file.
6. User1 now goes to PC3 where nobody has used the file rename trick,
and experiences the problem. Only solution is to play the rename trick
again in PC3.
So it seems you have to play the rename trick at least one per
workstation and that "fixes" the issue for any user who logs into that
workstation.
What other suggestions do you have? What debugging can I do next? I am
planning once everyone leaves the office today on changing the share
to bypass vfs gluster plugin and access the file directly from the
fuse mount, ie.:
# mount | grep export
10.0.1.7:/export on /export type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read= 131072)
Then set samba to share without using the vfs gluster plugin as follows
[Projects]
path = /export/projects
browseable = yes
write list = @Staff,root,@Admin,@Managers
writeable = yes
guest ok = no
create mask = 660
directory mask = 770
This test will determine if the issue is the samba vfs gluster plugin
or if it is the fact that the file is stored in the gluster volume.
Any other thoughts?
I suspect this has to do with locking and such.
Can create a new volume and share through vfs plugin.
Set logging to number 5 for samba and Gluster volume set client-log-level and server-log-level to TRACE.
Copy one rvt file.
Performs open, this is like fail.
Perform rename
Perform open, this would pass
Send us the logs.
I know it is troublesome but detailed logs are the only things that would help us analyze the issue.
Talur
Diego
On Wed, May 31, 2017 at 12:59 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
> The vfs gluster plugin for samba is linked against libglusterfs.so.0
> which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please see
> below.
>
> # ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
> libglusterfs.so.0 => /lib64/libglusterfs.so.0 (0x00007f61da858000)
>
> ]# yum provides /lib64/libglusterfs.so.0
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> * base: centos.vwtonline.net
> * extras: mirror.cs.vt.edu
> * updates: centosv.centos.org
> glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
> Repo : @centos-gluster310
> Matched from:
> Filename : /lib64/libglusterfs.so.0
>
>
>
> On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>> Samba is running in the same machine as glusterd. The machines were
>> rebooted after the upgrades and samba has been restarted a few times.
>>
>> # rpm -qa | grep gluster
>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>> glusterfs-server-3.10.2-1.el7.x86_64
>> glusterfs-api-3.10.2-1.el7.x86_64
>> glusterfs-3.10.2-1.el7.x86_64
>> glusterfs-cli-3.10.2-1.el7.x86_64
>> centos-release-gluster310-1.0-1.el7.centos.noarch
>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>> glusterfs-fuse-3.10.2-1.el7.x86_64
>> glusterfs-libs-3.10.2-1.el7.x86_64
>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>
>> # rpm -qa | grep samba
>> samba-common-libs-4.4.4-14.el7_3.x86_64
>> samba-common-tools-4.4.4-14.el7_3.x86_64
>> samba-libs-4.4.4-14.el7_3.x86_64
>> samba-4.4.4-14.el7_3.x86_64
>> samba-client-libs-4.4.4-14.el7_3.x86_64
>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>> samba-common-4.4.4-14.el7_3.noarch
>>
>> # cat /etc/redhat-release
>> CentOS Linux release 7.3.1611 (Core)
>>
>> I also raised the op version.
>>
>> # gluster volume get all cluster.op-version
>> Option Value
>> ------ -----
>> cluster.op-version 31000
>>
>> # gluster volume get all cluster.max-op-version
>> Option Value
>> ------ -----
>> cluster.max-op-version 31000
>>
>> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur <rtalur@xxxxxxxxxx> wrote:
>>> Also, please attach your smb.conf. You can directly attach in the list
>>> and need not have a google drive link.
>>>
>>>
>>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur <rtalur@xxxxxxxxxx> wrote:
>>>> Diego,
>>>>
>>>> I see that Samba is still using 3.6.6 Gluster. Is it possible you did
>>>> not restart smb after upgrading gluster(if samba is on same machine as
>>>> Gluster) or if you forgot to update Gluster client packages on Samba
>>>> node?
>>>>
>>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>>>>> Please download the log file from this link:
>>>>>
>>>>> https://drive.google.com/open?id= 0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>>>
>>>>> Let me know if you need any other log files.
>>>>>
>>>>> Diego
>>>>>
>>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur <rtalur@xxxxxxxxxx> wrote:
>>>>>> If possible please share the glusterfs-* log files from /var/log/samba.
>>>>>>
>>>>>> This might be because of cluster.lookup-optimize. Adding Poornima and
>>>>>> Raghavendra Gowdappa to help with this.
>>>>>>
>>>>>>
>>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>>>>>>> This is a bit puzzling, not sure what difference it would make, but:
>>>>>>>
>>>>>>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
>>>>>>> Revit opens and shows a window that says access denied.
>>>>>>>
>>>>>>> 2. Rename file, i.e from windows explorer right click, rename to
>>>>>>> MyRevitFile2.rvt
>>>>>>>
>>>>>>> 3. Without opening the file, rename file back to the original, i.e
>>>>>>> MyRevitFile.rvt
>>>>>>>
>>>>>>> 4. Double click on file and now it will open just fine without the
>>>>>>> Access Denied error.
>>>>>>>
>>>>>>> Any explanation for this? Could the rename operation be forcing or
>>>>>>> updating some attributes that then allow the file to open?
>>>>>>>
>>>>>>> Diego
>>>>>>>
>>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>>>>>>>> This is what I see in the logs set from smb.conf via line ->
>>>>>>>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>>>>>>>
>>>>>>>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>>>>>>>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
>>>>>>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>>>>>>>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>>>>>>> remote operation failed. Path:
>>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>>>>>>>> a directory]
>>>>>>>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>>>>>>> remote operation failed. Path:
>>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>>>>>>>> a directory]
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Over the weekend we updated a two server glusterfs 3.6.6 install to
>>>>>>>>> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
>>>>>>>>> enabled several of the newer caching features from gluster 3.9 for
>>>>>>>>> small file performance and samba, and we now seem to have some issues
>>>>>>>>> with accessing files from glusterfs. When users try to access some
>>>>>>>>> files, they get a Permission denied message. This seems to be only via
>>>>>>>>> samba as I am able to su - username and then do strings on the file.
>>>>>>>>>
>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster
>>>>>>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>>>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>>>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>>>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>>>>>
>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep samba
>>>>>>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>>>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>>>>>
>>>>>>>>> On the samba logs for the machine I notice something weird, samba
>>>>>>>>> seems to be trying to stat the file we are trying as a directory to
>>>>>>>>> see if it contains desktop.ini:
>>>>>>>>>
>>>>>>>>> [2017/05/30 10:13:07.297026, 0]
>>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_ stat)
>>>>>>>>> glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
>>>>>>>>> Not a directory
>>>>>>>>> [2017/05/30 10:13:07.298155, 0]
>>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_ stat)
>>>>>>>>> glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
>>>>>>>>> Not a directory
>>>>>>>>>
>>>>>>>>> This seems to be happening only with files with the .rvt extension.
>>>>>>>>> Though these files are usually larger in size vs other smaller excel,
>>>>>>>>> power point, etc files.
>>>>>>>>>
>>>>>>>>> Here are the complete options for the volume:
>>>>>>>>>
>>>>>>>>> https://pastebin.com/ZH2vMsMN
>>>>>>>>>
>>>>>>>>> I turned off performance.cache-samba-metadata again to see if that
>>>>>>>>> would help, but seems it does not help.
>>>>>>>>>
>>>>>>>>> I really appreciate any help with this.
>>>>>>>>>
>>>>>>>>> DIego
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users@xxxxxxxxxxx
>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
