Hi,
I was wondering if you have had a chance to review the logs and if you
can tell me know if more information is needed.
Diego
On Mon, Jun 12, 2017 at 8:20 AM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
> Did the logs provide any hints as to what the issue may be?
>
> Diego
>
> On Sat, Jun 3, 2017 at 12:16 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>> Thanks for taking the time to look into this. Since we needed downtime
>> due to the gluster update, we also updated the OS, including samba. We
>> went from 4.2.x to 4.4.4 and many other packages for CentOS were
>> updated as well. OS and samba updates were installed, then server
>> rebooted, then gluster was updated.
>>
>> Created a new test samba share to minimize logs, etc:
>>
>> [VfsGluster]
>> path = /vfsgluster
>> browseable = yes
>> write list = @Staff,root,@Admin,@Managers
>> writeable = yes
>> guest ok = no
>> create mask = 660
>> directory mask = 770
>> kernel share modes = No
>> vfs objects = glusterfs
>> glusterfs:loglevel = 7
>> glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
>> glusterfs:volume = export
>> veto files = desktop.ini
>>
>>
>> I am attaching the logs for the test of the rename operation. The file
>> I try to access is called:2016-9-13_Herrin Gear BMW_REV-3
>> The timeline is as follows (times are UTC):
>>
>> 15:18:00 -> Change smb.conf to glusterfs:loglevel = 9 and reload smbd
>> via smbcontrol smbd reload-config
>> See log: glusterfs-vfsgluster.log
>> 15:19:31 -> Adjust gluster client and brick log levels to TRACE
>> export.log is the Gluster volume log
>> bricks-hdds-brick.log is the Brick log on ysmha02 (did you need the
>> other one?) Samba is running on ysmah02.
>>
>> 15:20:40 -> Get the extended attributes for all files in the bricks on
>> both servers ysmha01 and ysmha02 before accessing the file:
>> cd /bricks/hdds/brick/vfsgluster/
>> find . -type f -exec getfattr -m . -d -e hex {} \; >>
>> /tmp/ysmha01-attributes-before-rename-run2
>> find . -type f -exec getfattr -m . -d -e hex {} \; >>
>> /tmp/ysmha02-attributes-before-rename-run2
>>
>> 15:21-ish -> Cannot open file so I do a rename and rename back via
>> Windows explorer
>> 15:22-ish -> Get the extended attirbutes again after renaming the
>> file. At this point I can open the file.
>> find . -type f -exec getfattr -m . -d -e hex {} \; >>
>> /tmp/ysmha01-attributes-after-rename-run2
>> find . -type f -exec getfattr -m . -d -e hex {} \; >>
>> /tmp/ysmha02-attributes-after-rename-run2
>>
>> I did all the files here, because Revit uses a file and some folders
>> to keep track of it changes, versions, etc.
>>
>> 15:23:25 -> Change gluster client and brick logs back to info.
>>
>> Let me know if I need to redo any tests, change any steps or provide
>> more logs I may have missed.
>>
>> Diego
>>
>> On Fri, Jun 2, 2017 at 8:19 PM, Raghavendra Talur <rtalur@xxxxxxxxxx> wrote:
>>>
>>>
>>> On 03-Jun-2017 3:27 AM, "Diego Remolina" <dijuremo@xxxxxxxxx> wrote:
>>>
>>> Hi everyone,
>>>
>>> Is there anything else we could do to check on this problem and try to
>>> fix it? The issue is definitively related to either the samba vfs
>>> gluster plugin or gluster itself. I am not sure how to pin it down
>>> futher.
>>>
>>>
>>> I don't think it is vfs plugin because you haven't updated samba packages
>>> and vfs plugin is same as before.
>>>
>>> It is either gfapi in Gluster or one of the lower xlators.
>>>
>>> Changing to fuse based shares might make the problem go away and can be used
>>> as a workaround.
>>>
>>>
>>>
>>> I went ahead and created a new share in the samba server which is on a
>>> local filesystem where the OS is installed, not part of gluster:
>>>
>>> # mount | grep home
>>> ]# ls -ld /home
>>> drwxr-xr-x. 14 root root 4096 Jun 2 17:21 /home
>>>
>>> This shows this is not mounted from anywhere. so I created a folder
>>> and shared it:
>>>
>>> ]# ls -ld /home/rvtsharetest/
>>> drwxrwx---. 3 dijuremo Staff 95 Jun 2 17:31 /home/rvtsharetest/
>>>
>>> ]# tail -7 /etc/samba/smb.conf
>>> [rvtsharetest]
>>> path=/home/rvtsharetest
>>> browseable = yes
>>> write list = @Staff,dijuremo
>>> guest ok = yes
>>> create mask = 664
>>> directory mask = 775
>>>
>>> When accessing Revit files in this new share the problem is *not* observed.
>>>
>>> When accessing Revit files on any of the samba shares that use the vfs
>>> gluster plugin and are stored in the gluster volume, we see the
>>> problems.
>>>
>>> Further analysis of the issue is even more disconcerting. As you may
>>> remember, I have found a workaround about renaming the file and back
>>> to the original, where things work. Here is where it gets more
>>> interesting. This seems to be workstation dependent, not user
>>> dependent.
>>>
>>> 1. User1 logs into PC1 and tries to access the file and gets the error
>>> message from Revit "ACCESS DENIED".
>>>
>>> 2. User1 uses windows explorer to go to the file location (we tried
>>> doing this from the server itself on the command line and it did not
>>> change anything, i.e. su - User1 then mv command). User1 renames the
>>> file, Revit.rvt -> Revit2.rvt, and clicks away for the rename to take
>>> place. User1 immediately renames the file to the original, Revit2.rvt
>>> -> Revit.rvt
>>>
>>> 3. User1 opens the file and everything works properly.
>>>
>>> 4. User2 logs into PC1 and tries to open the file. The file opens and
>>> works properly.
>>>
>>> 5. User2 logs into PC2 and tries to open the file, the problem comes
>>> up. User2 uses the rename trick and this fixes the problem on PC2.
>>> Even if User1 now comes to use PC2, User1 will have no problems with
>>> the file.
>>>
>>> 6. User1 now goes to PC3 where nobody has used the file rename trick,
>>> and experiences the problem. Only solution is to play the rename trick
>>> again in PC3.
>>>
>>> So it seems you have to play the rename trick at least one per
>>> workstation and that "fixes" the issue for any user who logs into that
>>> workstation.
>>>
>>> What other suggestions do you have? What debugging can I do next? I am
>>> planning once everyone leaves the office today on changing the share
>>> to bypass vfs gluster plugin and access the file directly from the
>>> fuse mount, ie.:
>>>
>>> # mount | grep export
>>> 10.0.1.7:/export on /export type fuse.glusterfs
>>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
>>>
>>> Then set samba to share without using the vfs gluster plugin as follows
>>>
>>> [Projects]
>>> path = /export/projects
>>> browseable = yes
>>> write list = @Staff,root,@Admin,@Managers
>>> writeable = yes
>>> guest ok = no
>>> create mask = 660
>>> directory mask = 770
>>>
>>> This test will determine if the issue is the samba vfs gluster plugin
>>> or if it is the fact that the file is stored in the gluster volume.
>>>
>>> Any other thoughts?
>>>
>>>
>>> I suspect this has to do with locking and such.
>>> Can create a new volume and share through vfs plugin.
>>> Set logging to number 5 for samba and Gluster volume set client-log-level
>>> and server-log-level to TRACE.
>>> Copy one rvt file.
>>> Performs open, this is like fail.
>>> Perform rename
>>> Perform open, this would pass
>>>
>>> Send us the logs.
>>>
>>> I know it is troublesome but detailed logs are the only things that would
>>> help us analyze the issue.
>>>
>>> Talur
>>>
>>>
>>> Diego
>>>
>>> On Wed, May 31, 2017 at 12:59 PM, Diego Remolina <dijuremo@xxxxxxxxx> wrote:
>>>> The vfs gluster plugin for samba is linked against libglusterfs.so.0
>>>> which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please see
>>>> below.
>>>>
>>>> # ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
>>>> libglusterfs.so.0 => /lib64/libglusterfs.so.0 (0x00007f61da858000)
>>>>
>>>> ]# yum provides /lib64/libglusterfs.so.0
>>>> Loaded plugins: fastestmirror
>>>> Loading mirror speeds from cached hostfile
>>>> * base: centos.vwtonline.net
>>>> * extras: mirror.cs.vt.edu
>>>> * updates: centosv.centos.org
>>>> glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
>>>> Repo : @centos-gluster310
>>>> Matched from:
>>>> Filename : /lib64/libglusterfs.so.0
>>>>
>>>>
>>>>
>>>> On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <dijuremo@xxxxxxxxx>
>>>> wrote:
>>>>> Samba is running in the same machine as glusterd. The machines were
>>>>> rebooted after the upgrades and samba has been restarted a few times.
>>>>>
>>>>> # rpm -qa | grep gluster
>>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>
>>>>> # rpm -qa | grep samba
>>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>
>>>>> # cat /etc/redhat-release
>>>>> CentOS Linux release 7.3.1611 (Core)
>>>>>
>>>>> I also raised the op version.
>>>>>
>>>>> # gluster volume get all cluster.op-version
>>>>> Option Value
>>>>> ------ -----
>>>>> cluster.op-version 31000
>>>>>
>>>>> # gluster volume get all cluster.max-op-version
>>>>> Option Value
>>>>> ------ -----
>>>>> cluster.max-op-version 31000
>>>>>
>>>>> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur <rtalur@xxxxxxxxxx>
>>>>> wrote:
>>>>>> Also, please attach your smb.conf. You can directly attach in the list
>>>>>> and need not have a google drive link.
>>>>>>
>>>>>>
>>>>>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur <rtalur@xxxxxxxxxx>
>>>>>> wrote:
>>>>>>> Diego,
>>>>>>>
>>>>>>> I see that Samba is still using 3.6.6 Gluster. Is it possible you did
>>>>>>> not restart smb after upgrading gluster(if samba is on same machine as
>>>>>>> Gluster) or if you forgot to update Gluster client packages on Samba
>>>>>>> node?
>>>>>>>
>>>>>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina <dijuremo@xxxxxxxxx>
>>>>>>> wrote:
>>>>>>>> Please download the log file from this link:
>>>>>>>>
>>>>>>>> https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>>>>>>
>>>>>>>> Let me know if you need any other log files.
>>>>>>>>
>>>>>>>> Diego
>>>>>>>>
>>>>>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur
>>>>>>>> <rtalur@xxxxxxxxxx> wrote:
>>>>>>>>> If possible please share the glusterfs-* log files from
>>>>>>>>> /var/log/samba.
>>>>>>>>>
>>>>>>>>> This might be because of cluster.lookup-optimize. Adding Poornima and
>>>>>>>>> Raghavendra Gowdappa to help with this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina <dijuremo@xxxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>>> This is a bit puzzling, not sure what difference it would make, but:
>>>>>>>>>>
>>>>>>>>>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
>>>>>>>>>> Revit opens and shows a window that says access denied.
>>>>>>>>>>
>>>>>>>>>> 2. Rename file, i.e from windows explorer right click, rename to
>>>>>>>>>> MyRevitFile2.rvt
>>>>>>>>>>
>>>>>>>>>> 3. Without opening the file, rename file back to the original, i.e
>>>>>>>>>> MyRevitFile.rvt
>>>>>>>>>>
>>>>>>>>>> 4. Double click on file and now it will open just fine without the
>>>>>>>>>> Access Denied error.
>>>>>>>>>>
>>>>>>>>>> Any explanation for this? Could the rename operation be forcing or
>>>>>>>>>> updating some attributes that then allow the file to open?
>>>>>>>>>>
>>>>>>>>>> Diego
>>>>>>>>>>
>>>>>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina
>>>>>>>>>> <dijuremo@xxxxxxxxx> wrote:
>>>>>>>>>>> This is what I see in the logs set from smb.conf via line ->
>>>>>>>>>>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>>>>>>>>>>
>>>>>>>>>>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>>>>>>>>>>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is
>>>>>>>>>>> NULL
>>>>>>>>>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>>>>>>>>>>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>>>>>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>>> Richmond/02
>>>>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini
>>>>>>>>>>> [Not
>>>>>>>>>>> a directory]
>>>>>>>>>>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>>>>>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>>>>>>>>>> remote operation failed. Path:
>>>>>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of
>>>>>>>>>>> Richmond/02
>>>>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini
>>>>>>>>>>> [Not
>>>>>>>>>>> a directory]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina
>>>>>>>>>>> <dijuremo@xxxxxxxxx> wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Over the weekend we updated a two server glusterfs 3.6.6 install
>>>>>>>>>>>> to
>>>>>>>>>>>> 3.10.2 We also updated samba and samba-vfs to the latest in
>>>>>>>>>>>> CentOS. I
>>>>>>>>>>>> enabled several of the newer caching features from gluster 3.9 for
>>>>>>>>>>>> small file performance and samba, and we now seem to have some
>>>>>>>>>>>> issues
>>>>>>>>>>>> with accessing files from glusterfs. When users try to access some
>>>>>>>>>>>> files, they get a Permission denied message. This seems to be only
>>>>>>>>>>>> via
>>>>>>>>>>>> samba as I am able to su - username and then do strings on the
>>>>>>>>>>>> file.
>>>>>>>>>>>>
>>>>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster
>>>>>>>>>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>>>>>>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>>>>>>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>>>>>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>>>>>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>>>>>>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>>>>>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>>>>>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>>>>>>>>
>>>>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep samba
>>>>>>>>>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>>>>>>>>
>>>>>>>>>>>> On the samba logs for the machine I notice something weird, samba
>>>>>>>>>>>> seems to be trying to stat the file we are trying as a directory
>>>>>>>>>>>> to
>>>>>>>>>>>> see if it contains desktop.ini:
>>>>>>>>>>>>
>>>>>>>>>>>> [2017/05/30 10:13:07.297026, 0]
>>>>>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>>> glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>>> failed:
>>>>>>>>>>>> Not a directory
>>>>>>>>>>>> [2017/05/30 10:13:07.298155, 0]
>>>>>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>>>>> glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini)
>>>>>>>>>>>> failed:
>>>>>>>>>>>> Not a directory
>>>>>>>>>>>>
>>>>>>>>>>>> This seems to be happening only with files with the .rvt
>>>>>>>>>>>> extension.
>>>>>>>>>>>> Though these files are usually larger in size vs other smaller
>>>>>>>>>>>> excel,
>>>>>>>>>>>> power point, etc files.
>>>>>>>>>>>>
>>>>>>>>>>>> Here are the complete options for the volume:
>>>>>>>>>>>>
>>>>>>>>>>>> https://pastebin.com/ZH2vMsMN
>>>>>>>>>>>>
>>>>>>>>>>>> I turned off performance.cache-samba-metadata again to see if that
>>>>>>>>>>>> would help, but seems it does not help.
>>>>>>>>>>>>
>>>>>>>>>>>> I really appreciate any help with this.
>>>>>>>>>>>>
>>>>>>>>>>>> DIego
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users@xxxxxxxxxxx
>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
