Re: missing files

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



That is very interesting. I tried this test and received a similar result. Start/stopping the volume causes a timing issue on the blank volume. It seems like there is some parameter getting set when you create a volume and gets reset when you start/stop a volume. Or, something gets set during the start/stop operation that causes the problem. Is there a way to list all parameters that are set for a volume? gluster volume info only shows the ones that the user has changed from defaults.

[root@gfs01bkp ~]# gluster volume stop test3brick
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: test3brick: success
[root@gfs01bkp ~]# gluster volume start test3brick
volume start: test3brick: success
[root@gfs01bkp ~]# mount /test3brick
[root@gfs01bkp ~]# cd /test3brick/
[root@gfs01bkp test3brick]# date; time tar -xPf boost_1_57_0.tar ; time rm -rf boost_1_57_0
Thu Feb 12 10:42:43 EST 2015

real    3m46.002s
user    0m0.421s
sys     0m2.812s

real    0m15.406s
user    0m0.092s
sys     0m0.549s


------ Original Message ------
From: "Pranith Kumar Karampuri" <pkarampu@xxxxxxxxxx>
To: "Justin Clift" <justin@xxxxxxxxxxx>; "Shyam" <srangana@xxxxxxxxxx>
Cc: "Gluster Devel" <gluster-devel@xxxxxxxxxxx>; "David F. Robinson" <david.robinson@xxxxxxxxxxxxx>
Sent: 2/12/2015 6:22:23 AM
Subject: Re:  missing files


On 02/12/2015 03:05 PM, Pranith Kumar Karampuri wrote:

On 02/12/2015 09:14 AM, Justin Clift wrote:
On 12 Feb 2015, at 03:02, Shyam <srangana@xxxxxxxxxx> wrote:
On 02/11/2015 08:28 AM, David F. Robinson wrote:
My base filesystem has 40-TB and the tar takes 19 minutes. I copied over 10-TB and it took the tar extraction from 1-minute to 7-minutes.

My suspicion is that it is related to number of files and not necessarily file size. Shyam is looking into reproducing this behavior on a redhat system.
I am able to reproduce the issue on a similar setup internally (at least at the surface it seems to be similar to what David is facing).

I will continue the investigation for the root cause.
Here is the initial analysis of my investigation: (Thanks for providing me with the setup shyam, keep the setup we may need it for further analysis)

On bad volume:
 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
 --------- ----------- ----------- ----------- ------------ ----
      0.00 0.00 us 0.00 us 0.00 us 937104 FORGET
      0.00 0.00 us 0.00 us 0.00 us 872478 RELEASE
      0.00 0.00 us 0.00 us 0.00 us 23668 RELEASEDIR
      0.00 41.86 us 23.00 us 86.00 us 92 STAT
      0.01 39.40 us 24.00 us 104.00 us 218 STATFS
      0.28 55.99 us 43.00 us 1152.00 us 4065 SETXATTR
      0.58 56.89 us 25.00 us 4505.00 us 8236 OPENDIR
      0.73 26.80 us 11.00 us 257.00 us 22238 FLUSH
      0.77 152.83 us 92.00 us 8819.00 us 4065 RMDIR
      2.57 62.00 us 21.00 us 409.00 us 33643 WRITE
      5.46 199.16 us 108.00 us 469938.00 us 22238 UNLINK
      6.70 69.83 us 43.00 us 7777.00 us 77809 LOOKUP
      6.97 447.60 us 21.00 us 54875.00 us 12631 READDIRP
      7.73 79.42 us 33.00 us 1535.00 us 78909 SETATTR
     14.11 2815.00 us 176.00 us 2106305.00 us 4065 MKDIR
     54.09 1972.62 us 138.00 us 1520773.00 us 22238 CREATE

On good volume:
 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
 --------- ----------- ----------- ----------- ------------ ----
      0.00 0.00 us 0.00 us 0.00 us 58870 FORGET
      0.00 0.00 us 0.00 us 0.00 us 66016 RELEASE
      0.00 0.00 us 0.00 us 0.00 us 16480 RELEASEDIR
      0.00 61.50 us 58.00 us 65.00 us 2 OPEN
      0.01 39.56 us 16.00 us 112.00 us 71 STAT
      0.02 41.29 us 27.00 us 79.00 us 163 STATFS
      0.03 36.06 us 17.00 us 98.00 us 301 FSTAT
      0.79 62.38 us 39.00 us 269.00 us 4065 SETXATTR
      1.14 242.99 us 25.00 us 28636.00 us 1497 READ
      1.54 59.76 us 25.00 us 6325.00 us 8236 OPENDIR
      1.70 133.75 us 89.00 us 374.00 us 4065 RMDIR
      2.25 32.65 us 15.00 us 265.00 us 22006 FLUSH
      3.37 265.05 us 172.00 us 2349.00 us 4065 MKDIR
      7.14 68.34 us 21.00 us 21902.00 us 33357 WRITE
     11.00 159.68 us 107.00 us 2567.00 us 22003 UNLINK
     13.82 200.54 us 133.00 us 21762.00 us 22003 CREATE
     17.85 448.85 us 22.00 us 54046.00 us 12697 READDIRP
     18.37 76.12 us 45.00 us 294.00 us 77044 LOOKUP
     20.95 85.54 us 35.00 us 1404.00 us 78204 SETATTR

As we can see here, FORGET/RELEASE are way more in the brick from full volume compared to the brick from empty volume. It seems to suggest that the inode-table on the volume with lots of data is carrying too many passive inodes in the table which need to be displaced to create new ones. Need to check if they come in the fop-path. Need to continue my investigations further, will let you know.
Just to increase confidence performed one more test. Stopped the volumes and re-started. Now on both the volumes, the numbers are almost same:

[root@gqac031 gluster-mount]# time rm -rf boost_1_57_0 ; time tar xf boost_1_57_0.tar.gz

real 1m15.074s
user 0m0.550s
sys 0m4.656s

real 2m46.866s
user 0m5.347s
sys 0m16.047s

[root@gqac031 gluster-mount]# cd /gluster-emptyvol/
[root@gqac031 gluster-emptyvol]# ls
boost_1_57_0.tar.gz
[root@gqac031 gluster-emptyvol]# time tar xf boost_1_57_0.tar.gz

real 2m31.467s
user 0m5.475s
sys 0m15.471s

gqas015.sbu.lab.eng.bos.redhat.com:testvol on /gluster-mount type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) gqas015.sbu.lab.eng.bos.redhat.com:emotyvol on /gluster-emptyvol type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

Pranith

Pranith
Thanks Shyam. :)

+ Justin

-- GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel




[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux