Did the logs provide any hints as to what the issue may be?
Diego
On Sat, Jun 3, 2017 at 12:16 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
> Thanks for taking the time to look into this. Since we needed downtime
> due to the gluster update, we also updated the OS, including samba. We
> went from 4.2.x to 4.4.4 and many other packages for CentOS were
> updated as well. OS and samba updates were installed, then server
> rebooted, then gluster was updated.
>
> Created a new test samba share to minimize logs, etc:
>
> [VfsGluster]
> path = /vfsgluster
> browseable = yes
> write list = @Staff,root,@Admin,@Managers
> writeable = yes
> guest ok = no
> create mask = 660
> directory mask = 770
> kernel share modes = No
> vfs objects = glusterfs
> glusterfs:loglevel = 7
> glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
> glusterfs:volume = export
> veto files = desktop.ini
>
>
> I am attaching the logs for the test of the rename operation. The file
> I try to access is called:2016-9-13_Herrin Gear BMW_REV-3
> The timeline is as follows (times are UTC):
>
> 15:18:00 -> Change smb.conf to glusterfs:loglevel = 9 and reload smbd
> via smbcontrol smbd reload-config
> See log: glusterfs-vfsgluster.log
> 15:19:31 -> Adjust gluster client and brick log levels to TRACE
> export.log is the Gluster volume log
> bricks-hdds-brick.log is the Brick log on ysmha02 (did you need the
> other one?) Samba is running on ysmah02.
>
> 15:20:40 -> Get the extended attributes for all files in the bricks on
> both servers ysmha01 and ysmha02 before accessing the file:
> cd /bricks/hdds/brick/vfsgluster/
> find . -type f -exec getfattr -m . -d -e hex {} \; >>
> /tmp/ysmha01-attributes-before-rename-run2
> find . -type f -exec getfattr -m . -d -e hex {} \; >>
> /tmp/ysmha02-attributes-before-rename-run2
>
> 15:21-ish -> Cannot open file so I do a rename and rename back via
> Windows explorer
> 15:22-ish -> Get the extended attirbutes again after renaming the
> file. At this point I can open the file.
> find . -type f -exec getfattr -m . -d -e hex {} \; >>
> /tmp/ysmha01-attributes-after-rename-run2
> find . -type f -exec getfattr -m . -d -e hex {} \; >>
> /tmp/ysmha02-attributes-after-rename-run2
>
> I did all the files here, because Revit uses a file and some folders
> to keep track of it changes, versions, etc.
>
> 15:23:25 -> Change gluster client and brick logs back to info.
>
> Let me know if I need to redo any tests, change any steps or provide
> more logs I may have missed.
>
> Diego
>
> On Fri, Jun 2, 2017 at 8:19 PM, Raghavendra Talur <rtalur@xxxxxxxxxx> wrote:
>>
>>
>> On 03-Jun-2017 3:27 AM, "Diego Remolina" <dijuremo@xxxxxxxxx> wrote:
>>
>> Hi everyone,
>>
>> Is there anything else we could do to check on this problem and try to
>> fix it? The issue is definitively related to either the samba vfs
>> gluster plugin or gluster itself. I am not sure how to pin it down
>> futher.
>>
>>
>> I don't think it is vfs plugin because you haven't updated samba packages
>> and vfs plugin is same as before.
>>
>> It is either gfapi in Gluster or one of the lower xlators.
>>
>> Changing to fuse based shares might make the problem go away and can be used
>> as a workaround.
>>
>>
>>
>> I went ahead and created a new share in the samba server which is on a
>> local filesystem where the OS is installed, not part of gluster:
>>
>> # mount | grep home
>> ]# ls -ld /home
>> drwxr-xr-x. 14 root root 4096 Jun 2 17:21 /home
>>
>> This shows this is not mounted from anywhere. so I created a folder
>> and shared it:
>>
>> ]# ls -ld /home/rvtsharetest/
>> drwxrwx---. 3 dijuremo Staff 95 Jun 2 17:31 /home/rvtsharetest/
>>
>> ]# tail -7 /etc/samba/smb.conf
>> [rvtsharetest]
>> path=/home/rvtsharetest
>> browseable = yes
>> write list = @Staff,dijuremo
>> guest ok = yes
>> create mask = 664
>> directory mask = 775
>>
>> When accessing Revit files in this new share the problem is *not* observed.
>>
>> When accessing Revit files on any of the samba shares that use the vfs
>> gluster plugin and are stored in the gluster volume, we see the
>> problems.
>>
>> Further analysis of the issue is even more disconcerting. As you may
>> remember, I have found a workaround about renaming the file and back
>> to the original, where things work. Here is where it gets more
>> interesting. This seems to be workstation dependent, not user
>> dependent.
>>
>> 1. User1 logs into PC1 and tries to access the file and gets the error
>> message from Revit "ACCESS DENIED".
>>
>> 2. User1 uses windows explorer to go to the file location (we tried
>> doing this from the server itself on the command line and it did not
>> change anything, i.e. su - User1 then mv command). User1 renames the
>> file, Revit.rvt -> Revit2.rvt, and clicks away for the rename to take
>> place. User1 immediately renames the file to the original, Revit2.rvt
>> -> Revit.rvt
>>
>> 3. User1 opens the file and everything works properly.
>>
>> 4. User2 logs into PC1 and tries to open the file. The file opens and
>> works properly.
>>
>> 5. User2 logs into PC2 and tries to open the file, the problem comes
>> up. User2 uses the rename trick and this fixes the problem on PC2.
>> Even if User1 now comes to use PC2, User1 will have no problems with
>> the file.
>>
>> 6. User1 now goes to PC3 where nobody has used the file rename trick,
>> and experiences the problem. Only solution is to play the rename trick
>> again in PC3.
>>
>> So it seems you have to play the rename trick at least one per
>> workstation and that "fixes" the issue for any user who logs into that
>> workstation.
>>
>> What other suggestions do you have? What debugging can I do next? I am
>> planning once everyone leaves the office today on changing the share
>> to bypass vfs gluster plugin and access the file directly from the
>> fuse mount, ie.:
>>
>> # mount | grep export
>> 10.0.1.7:/export on /export type fuse.glusterfs
>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
>>
>> Then set samba to share without using the vfs gluster plugin as follows
>>
>> [Projects]
>> path = /export/projects
>> browseable = yes
>> write list = @Staff,root,@Admin,@Managers
>> writeable = yes
>> guest ok = no
>> create mask = 660
>> directory mask = 770
>>
>> This test will determine if the issue is the samba vfs gluster plugin
>> or if it is the fact that the file is stored in the gluster volume.
>>
>> Any other thoughts?
>>
>>
>> I suspect this has to do with locking and such.
>> Can create a new volume and share through vfs plugin.
>> Set logging to number 5 for samba and Gluster volume set client-log-level
>> and server-log-level to TRACE.
>> Copy one rvt file.
>> Performs open, this is like fail.
>> Perform rename
>> Perform open, this would pass
>>
>> Send us the logs.
>>
>> I know it is troublesome but detailed logs are the only things that would
>> help us analyze the issue.
>>
>> Talur
>>
>>
>> Diego
>>
>> On Wed, May 31, 2017 at 12:59 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>>> The vfs gluster plugin for samba is linked against libglusterfs.so.0
>>> which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please see
>>> below.
>>>
>>> # ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
>>> libglusterfs.so.0 => /lib64/libglusterfs.so.0 (0x00007f61da858000)
>>>
>>> ]# yum provides /lib64/libglusterfs.so.0
>>> Loaded plugins: fastestmirror
>>> Loading mirror speeds from cached hostfile
>>> * base: centos.vwtonline.net
>>> * extras: mirror.cs.vt.edu
>>> * updates: centosv.centos.org
>>> glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
>>> Repo : @centos-gluster310
>>> Matched from:
>>> Filename : /lib64/libglusterfs.so.0
>>>
>>>
>>>
>>> On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <dijuremo@xxxxxxxxx>
>>> wrote:
>>>> Samba is running in the same machine as glusterd. The machines were
>>>> rebooted after the upgrades and samba has been restarted a few times.
>>>>
>>>> # rpm -qa | grep gluster
>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>> glusterfs-3.10.2-1.el7.x86_64
>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>
>>>> # rpm -qa | grep samba
>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>> samba-4.4.4-14.el7_3.x86_64
>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>
>>>> # cat /etc/redhat-release
>>>> CentOS Linux release 7.3.1611 (Core)
>>>>
>>>> I also raised the op version.
>>>>
>>>> # gluster volume get all cluster.op-version
>>>> Option Value
>>>> ------ -----
>>>> cluster.op-version 31000
>>>>
>>>> # gluster volume get all cluster.max-op-version
>>>> Option Value
>>>> ------ -----
>>>> cluster.max-op-version 31000
>>>>
>>>> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur <rtalur@xxxxxxxxxx>
>>>> wrote:
>>>>> Also, please attach your smb.conf. You can directly attach in the list
>>>>> and need not have a google drive link.
>>>>>
>>>>>
>>>>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur <rtalur@xxxxxxxxxx>
>>>>> wrote:
>>>>>> Diego,
>>>>>>
>>>>>> I see that Samba is still using 3.6.6 Gluster. Is it possible you did
>>>>>> not restart smb after upgrading gluster(if samba is on same machine as
>>>>>> Gluster) or if you forgot to update Gluster client packages on Samba
>>>>>> node?
>>>>>>
>>>>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina <dijuremo@xxxxxxxxx>
>>>>>> wrote:
>>>>>>> Please download the log file from this link:
>>>>>>>
>>>>>>> https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>>>>>
>>>>>>> Let me know if you need any other log files.
>>>>>>>
>>>>>>> Diego
>>>>>>>
>>>>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur
>>>>>>> <rtalur@xxxxxxxxxx> wrote:
>>>>>>>> If possible please share the glusterfs-* log files from
>>>>>>>> /var/log/samba.
>>>>>>>>
>>>>>>>> This might be because of cluster.lookup-optimize. Adding Poornima and
>>>>>>>> Raghavendra Gowdappa to help with this.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina <dijuremo@xxxxxxxxx>
>>>>>>>> wrote:
>>>>>>>>> This is a bit puzzling, not sure what difference it would make, but:
>>>>>>>>>
>>>>>>>>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
>>>>>>>>> Revit opens and shows a window that says access denied.
>>>>>>>>>
>>>>>>>>> 2. Rename file, i.e from windows explorer right click, rename to
>>>>>>>>> MyRevitFile2.rvt
>>>>>>>>>
>>>>>>>>> 3. Without opening the file, rename file back to the original, i.e
>>>>>>>>> MyRevitFile.rvt
>>>>>>>>>
>>>>>>>>> 4. Double click on file and now it will open just fine without the
>>>>>>>>> Access Denied error.
>>>>>>>>>
>>>>>>>>> Any explanation for this? Could the rename operation be forcing or
>>>>>>>>> updating some attributes that then allow the file to open?
>>>>>>>>>
>>>>>>>>> Diego
>>>>>>>>>
>>>>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina
>>>>>>>>> <dijuremo@xxxxxxxxx> wrote:
>>>>>>>>>> This is what I see in the logs set from smb.conf via line ->
>>>>>>>>>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>>>>>>>>>
>>>>>>>>>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>>>>>>>>>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is
>>>>>>>>>> NULL
>>>>>>>>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>>>>>>>>>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>>>>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>> Richmond/02
>>>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini
>>>>>>>>>> [Not
>>>>>>>>>> a directory]
>>>>>>>>>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>>>>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>> Richmond/02
>>>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini
>>>>>>>>>> [Not
>>>>>>>>>> a directory]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina
>>>>>>>>>> <dijuremo@xxxxxxxxx> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Over the weekend we updated a two server glusterfs 3.6.6 install
>>>>>>>>>>> to
>>>>>>>>>>> 3.10.2 We also updated samba and samba-vfs to the latest in
>>>>>>>>>>> CentOS. I
>>>>>>>>>>> enabled several of the newer caching features from gluster 3.9 for
>>>>>>>>>>> small file performance and samba, and we now seem to have some
>>>>>>>>>>> issues
>>>>>>>>>>> with accessing files from glusterfs. When users try to access some
>>>>>>>>>>> files, they get a Permission denied message. This seems to be only
>>>>>>>>>>> via
>>>>>>>>>>> samba as I am able to su - username and then do strings on the
>>>>>>>>>>> file.
>>>>>>>>>>>
>>>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster
>>>>>>>>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>>>>>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>>>>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>>>>>>>
>>>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep samba
>>>>>>>>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>>>>>>>
>>>>>>>>>>> On the samba logs for the machine I notice something weird, samba
>>>>>>>>>>> seems to be trying to stat the file we are trying as a directory
>>>>>>>>>>> to
>>>>>>>>>>> see if it contains desktop.ini:
>>>>>>>>>>>
>>>>>>>>>>> [2017/05/30 10:13:07.297026, 0]
>>>>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>> glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>> failed:
>>>>>>>>>>> Not a directory
>>>>>>>>>>> [2017/05/30 10:13:07.298155, 0]
>>>>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>> glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>> failed:
>>>>>>>>>>> Not a directory
>>>>>>>>>>>
>>>>>>>>>>> This seems to be happening only with files with the .rvt
>>>>>>>>>>> extension.
>>>>>>>>>>> Though these files are usually larger in size vs other smaller
>>>>>>>>>>> excel,
>>>>>>>>>>> power point, etc files.
>>>>>>>>>>>
>>>>>>>>>>> Here are the complete options for the volume:
>>>>>>>>>>>
>>>>>>>>>>> https://pastebin.com/ZH2vMsMN
>>>>>>>>>>>
>>>>>>>>>>> I turned off performance.cache-samba-metadata again to see if that
>>>>>>>>>>> would help, but seems it does not help.
>>>>>>>>>>>
>>>>>>>>>>> I really appreciate any help with this.
>>>>>>>>>>>
>>>>>>>>>>> DIego
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users@xxxxxxxxxxx
>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
