Re: 3.6.1 issue

["David F. Robinson" <david.robinson@xxxxxxxxxxxxx>]

That did not fix the issue (see below).  I also have run into another possibly related issue.  After untarring the boost directory and compiling the software, I cannot delete the source directory structure.  It says "direct not empty".

corvidpost5:temp3/gfs> \rm -r boost_1_57_0
rm: cannot remove `boost_1_57_0/libs/numeric/odeint/test': Directory not empty
corvidpost5:temp3/gfs> cd boost_1_57_0/libs/numeric/odeint/test/
corvidpost5:odeint/test> ls -al
total 0
drwxr-x--- 3 dfrobins users  94 Dec 20 01:51 ./
drwx------ 3 dfrobins users 100 Dec 20 01:51 ../

 

 

 

cluster.read-hash-mode to 2 results 

 

corvidpost5:TankExamples/DakotaList> ls -al
total 5
drwxr-x--- 2 dfrobins users  166 Dec 22 11:16 ./
drwxr-x--- 6 dfrobins users  445 Dec 22 11:16 ../
lrwxrwxrwx 1 dfrobins users   25 Dec 22 11:16 EvalTank.py -> ../tank_model/EvalTank.py*
---------- 1 dfrobins users    0 Dec 22 11:16 FEMTank.py
-rwx--x--- 1 dfrobins users  734 Nov  7 11:05 RunTank.sh*
-rw------- 1 dfrobins users 1432 Nov  7 11:05 dakota_PandL_list.in
-rw------- 1 dfrobins users 1860 Nov  7 11:05 dakota_Ponly_list.in
 

gluster> volume info homegfs
 
Volume Name: homegfs
Type: Distributed-Replicate
Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs
Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs
Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs
Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs
Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs
Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs
Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs
Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs
Options Reconfigured:
cluster.read-hash-mode: 2
performance.stat-prefetch: off
performance.io-thread-count: 32
performance.cache-size: 128MB
performance.write-behind-window-size: 128MB
server.allow-insecure: on
network.ping-timeout: 10
storage.owner-gid: 100
geo-replication.indexing: off
geo-replication.ignore-pid-check: on
changelog.changelog: on
changelog.fsync-interval: 3
changelog.rollover-time: 15
server.manage-gids: on
 

 

 

------ Original Message ------

From: "Vijay Bellur" <vbellur@xxxxxxxxxx>

To: "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>

Cc: "Justin Clift" <justin@xxxxxxxxxxx>; "Gluster Devel" <gluster-devel@xxxxxxxxxxx>

Sent: 12/22/2014 9:23:44 AM

Subject: Re: 3.6.1 issue

 

On 12/21/2014 11:10 PM, David F. Robinson wrote:

So for now it is up to all of the individual users to know they cannot use tar without the -P switch if they are accessing a data storage system that uses gluster?

 

 

Setting volume option cluster.read-hash-mode to 2 could help here. Can you please check if this resolves the problem without -P switch?

 

-Vijay

 

 

On Dec 21, 2014, at 12:30 PM, Vijay Bellur <vbellur@xxxxxxxxxx> wrote:

 

On 12/20/2014 12:09 PM, David F. Robinson wrote:

Seems to work with "-xPf". I obviously couldn't check all of the files,

but the two specific ones that I noted in my original email do not show

any problems when using -P...

 

This is related to the way tar extracts symbolic links by default & its interaction with GlusterFS. In a nutshell the following steps are involved in creation of symbolic links on the destination:

 

a) Create an empty regular placeholder file with permission bits set to 0 and the name being that of the symlink source file.

 

b) Record the device, inode numbers and the mtime of the placeholder file through stat.

 

c) After the first pass of extraction is complete, there is a second pass involved to set right symbolic links. In this phase a stat is performed on the placeholder file. If all attributes recorded in b) are in sync with the latest information from stat buf, only then the placeholder is unlinked and a new symbolic link is created. If any attribute is out of sync, the unlink and creation of symbolic link do not happen.

 

In the case of replicated GlusterFS volumes, the mtimes can vary across nodes during the creation of placeholder files. If the stat calls in steps b) and c) land on different nodes, then there is a very good likelihood that tar would skip creation of symbolic links and leave behind the placeholder files.

 

A little more detail about this particular implementation behavior of symlinks for tar can be found at [1].

 

To overcome this behavior, we can make use of the P switch with tar command during extraction which will create the link file directly and not go ahead with the above set of steps.

 

Keeping timestamps in sync across the cluster will help to an extent in preventing this situation. There are ongoing refinements in replicate's selection of read-child which will help in addressing this problem.

 

-Vijay

 

[1] http://lists.debian.org/debian-user/2003/03/msg03249.html

 

 

 

 

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel