Patch 354 is most responsible for the speed up. See data:
Patch 353
real 6m40.655s
user 0m0.120s
sys 0m1.380s
Patch 354
real 2m56.825s
user 0m0.130s
sys 0m1.400s
Patch 362
real 2m54.541s
user 0m0.100s
sys 0m1.220s
Harris
----- Original Message -----
From: "Harris Landgarten" <harrisl@xxxxxxxxxxxxx>
To: "Harris Landgarten" <harrisl@xxxxxxxxxxxxx>
Cc: "gluster-devel" <gluster-devel@xxxxxxxxxx>, "Anand Avati" <avati@xxxxxxxxxxxxx>
Sent: Wednesday, July 25, 2007 9:05:56 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
Testing with Patch 361
real 2m58.516s
user 0m0.140s
sys 0m1.470s
Unbelievable.
Harris
----- Original Message -----
From: "Harris Landgarten" <harrisl@xxxxxxxxxxxxx>
To: "Harris Landgarten" <harrisl@xxxxxxxxxxxxx>
Cc: "gluster-devel" <gluster-devel@xxxxxxxxxx>, "Anand Avati" <avati@xxxxxxxxxxxxx>
Sent: Monday, July 23, 2007 9:58:36 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
I have narrowed the problem down to Patch-333:
Patch 332
real 6m37.204s
user 0m0.180s
sys 0m1.590s
Patch 333
real 10m38.697s
user 0m0.110s
sys 0m1.300s
Patch 334
real 10m47.733s
user 0m0.190s
sys 0m1.570s
Patch 347
real 10m20.522s
user 0m0.130s
sys 0m1.590s
Patch-333 introduced some changes to readahead to handle atimes. My configs haven't changed and atime updates are not occurring but the slow down is dramatic.
Harris
----- Original Message -----
From: "Harris Landgarten" <harrisl@xxxxxxxxxxxxx>
To: "Anand Avati" <avati@xxxxxxxxxxxxx>
Cc: "gluster-devel" <gluster-devel@xxxxxxxxxx>
Sent: Sunday, July 22, 2007 8:33:56 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
# du -h 0 1 2 3
134M 0
79M 1
121M 2
151M 3
# ls -lh /mnt/glusterfs/test/test.tbz
-rw-r--r-- 1 root root 460M Jul 21 20:04 /mnt/glusterfs/test/test.tbz
Harris
----- Original Message -----
From: "Anand Avati" <avati@xxxxxxxxxxxxx>
To: "Harris Landgarten" <harrisl@xxxxxxxxxxxxx>
Cc: "Amar S. Tumballi" <amar@xxxxxxxxxxxxx>, "gluster-devel" <gluster-devel@xxxxxxxxxx>
Sent: Sunday, July 22, 2007 8:26:21 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
approximate combined size of 0 1 2 and 3 ?
avati
2007/7/22 , Harris Landgarten < harrisl@xxxxxxxxxxxxx >:
The 0 1 2 3 folders are on the same glusterfs mount. This is a test of gluster to gluster tar. The 0 1 2 3 folders are part of a mail server store tree. The resulting tar contains 15547 objects ranging in size from 100b to 10m.
Harris
----- Original Message -----
From: "Anand Avati" < avati@xxxxxxxxxxxxx >
To: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
Cc: "Amar S. Tumballi" < amar@xxxxxxxxxxxxx >, "gluster-devel" < gluster-devel@xxxxxxxxxx >
Sent: Sunday, July 22, 2007 8:08:17 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
Harris,
where are the 0 1 and 2 files/dirs? on the same glusterfs mount or on local disk?
avati
2007/7/22 , Harris Landgarten < harrisl@xxxxxxxxxxxxx >:
Nothing in spec about flush-behind.
### Add writeback feature
volume writeback
type performance/write-behind
option aggregate-size 131072 # unit in bytes
subvolumes bricks
end-volume
Harris
----- Original Message -----
From: "Anand Avati" < avati@xxxxxxxxxxxxx >
To: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
Cc: "Amar S. Tumballi" < amar@xxxxxxxxxxxxx >, "gluster-devel" < gluster-devel@xxxxxxxxxx >
Sent: Sunday, July 22, 2007 2:29:08 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
Harris,
do you have 'option flush-behind on' set in the write-behind section of your client spec file?
avati
2007/7/22 , Harris Landgarten < harrisl@xxxxxxxxxxxxx >:
There is a major slow down in patch 336 and beyond. Here are some numbers:
Patch 331
time tar -cvf /mnt/glusterfs/test/test.tbz 0 1 2 3
real 6m36.388s
user 0m0.150s
sys 0m1.400s
Patch 336
time tar -cvf /mnt/glusterfs/test/test.tbz 0 1 2 3
real 11m5.022s
user 0m0.180s
sys 0m1.420s
Patch 341
time tar -cvf /mnt/glusterfs/test/test.tbz 0 1 2 3
real 12m8.700s
user 0m0.170s
sys 0m1.550s
Patch 344
time tar -cvf /mnt/glusterfs/test/test.tbz 0 1 2 3
real 11m10.577s
user 0m0.130s
sys 0m1.700s
Something in patch 332-336 seems to be the problem.
Harris
----- Original Message -----
From: "Amar S. Tumballi" < amar@xxxxxxxxxxxxx >
To: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
Cc: "Anand Avati" < avati@xxxxxxxxxxxxx >, "gluster-devel" < gluster-devel@xxxxxxxxxx >
Sent: Friday, July 20, 2007 7:17:56 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
Hi Harris,
Thanks for notifying us. Fix committed. (patch 331)
-amar
On 7/20/07 , Harris Landgarten < harrisl@xxxxxxxxxxxxx > wrote:
Strange crash.
After restarting client, df -h crashes. If ls /mnt/glusterfs is run first, df -h runs fine. Verified on both clients.
Here is the bt from Ubuntu
Program terminated with signal 11, Segmentation fault.
#0 data_to_ptr (data=0x8091e30) at dict.c:812
812 {
(gdb) bt
#0 data_to_ptr (data=0x8091e30) at dict.c:812
#1 0xb7faacc6 in default_statfs (frame=0x8091e30, this=0x80586c0, loc=0x8091cc4) at defaults.c:1001
#2 0xb7faacc6 in default_statfs (frame=0x8091da0, this=0x8058760, loc=0x8091cc4) at defaults.c:1001
#3 0x0804c19b in fuse_statfs (req=0x8091c68, ino=0) at fuse-bridge.c:1496
#4 0xb7f9ab12 in fuse_reply_statfs_compat () from /usr/lib/libfuse.so.2
#5 0xb7f9b1e3 in fuse_reply_entry () from /usr/lib/libfuse.so.2
#6 0xb7f9c9c6 in fuse_session_process () from /usr/lib/libfuse.so.2
#7 0x0804e019 in fuse_transport_notify (xl=0x8058c90, event=2, data=0x8052408) at fuse-bridge.c:2028
#8 0xb7fad7c7 in transport_notify (this=0x8057a7c, event=0) at transport.c:152
#9 0xb7fae239 in sys_epoll_iteration (ctx=0xbfecfec4) at epoll.c:54
#10 0xb7fad89d in poll_iteration (ctx=0xbfecfec4) at transport.c:260
#11 0x0804a36b in main (argc=6, argv=0xbfecffa4) at glusterfs.c:382
Harris
----- Original Message -----
From: "Anand Avati" < avati@xxxxxxxxxxxxx >
To: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
Cc: "Amar S. Tumballi" < amar@xxxxxxxxxxxxx >, "gluster-devel" < gluster-devel@xxxxxxxxxx >
Sent: Thursday, July 19, 2007 4:17:39 PM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
Harris,
both the slowness bug and the segfault you reported have been fixed in the latest TLA patchset. Please update and confirm that the fixes work for you.
thanks,
avati
2007/7/17 , Harris Landgarten < harrisl@xxxxxxxxxxxxx >:
Amar,
Anything on this bug. If you can tell me the tla command to get specific patch levels I will try to narrow to bug down further.
Harris
----- Original Message -----
From: "Amar S. Tumballi" < amar@xxxxxxxxxxxxx >
To: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
Cc: "gluster-devel" < gluster-devel@xxxxxxxxxx >
Sent: Monday, July 16, 2007 12:31:19 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
Hi Harris,
Thanks for cornering the bugs between 309-313 . We are looking into it.
-amar
On 7/15/07 , Harris Landgarten < harrisl@xxxxxxxxxxxxx > wrote:
I just tested a full zimbra backup on my mailbox with 308 client and 313 bricks and it completed in normal time with no errors. This is more evidence that the problem is client only and was introduced in 309-313 .
Harris
----- Original Message -----
From: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
To: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
Cc: "gluster-devel" < gluster-devel@xxxxxxxxxx >
Sent: Sunday, July 15, 2007 8:30:15 AM (GMT-0500) America/New_York
Subject: Re: Slowness and segfault with 313
Some more testing:
Patch 308
time tar -cvf /mnt/glusterfs/test/test.tbz 0 1 2 3
real 6m47.947s
user 0m0.180s
sys 0m1.220s
Patch 313
time tar -cvf /mnt/glusterfs/test/test.tbz 0 1 2 3
real 9m21.909s
user 0m0.160s
sys 0m1.470s
Patch 313 also used 50% more memory.
This leads me to suspect the problem is in writebehind or posix-locks
Harris
----- Original Message -----
From: "Harris Landgarten" < harrisl@xxxxxxxxxxxxx >
To: "gluster-devel" < gluster-devel@xxxxxxxxxx >
Sent: Saturday, July 14, 2007 2:23:20 PM (GMT-0500) America/New_York
Subject: Slowness and segfault with 313
Last weekend with 299 a full Zimbra backup completed in 27 minutes. This weekend with 313 the backup was only about 1/2 through after 5 hrs. I aborted the backup and the client crashed. The abort would have tried to remove about 4G of files from the /mnt/glusterfs/backups/tmp folder. The following BT was generated from the core:
Core was generated by `[glusterfs] '.
Program terminated with signal 11, Segmentation fault.
#0 unify_unlink (frame=0xdec4b30, this=0x80585b0, loc=0xddb98dc) at unify.c:2256
2256 list = loc->inode->private;
(gdb) bt
#0 unify_unlink (frame=0xdec4b30, this=0x80585b0, loc=0xddb98dc) at unify.c:2256
#1 0xb7f32c66 in default_unlink (frame=0xdbc51f8, this=0x80592c0, loc=0xddb98dc) at defaults.c:480
#2 0xb7f32c66 in default_unlink (frame=0xde06ca8, this=0x8059350, loc=0xddb98dc) at defaults.c:480
#3 0x0804ccf3 in fuse_unlink (req=0xdf68980, par= 4786914 , name=0xcc16770 "BbZslzxnS2Y,8Az0m2v5ExLBXbs= 6411-6246.msg1") at fuse-bridge.c:781
#4 0xb7f21461 in fuse_reply_err () from /usr/lib/libfuse.so.2
#5 0xb7f221e3 in fuse_reply_entry () from /usr/lib/libfuse.so.2
#6 0xb7f239c6 in fuse_session_process () from /usr/lib/libfuse.so.2
#7 0x0804abae in fuse_transport_notify (xl=0x8059910, event=2, data=0x8053410) at fuse-bridge.c:1942
#8 0xb7f34cc7 in transport_notify (this=0xddb98dc, event= 204699600 ) at transport.c:152
#9 0xb7f35979 in sys_epoll_iteration (ctx=0xbfb56b14) at epoll.c:54
#10 0xb7f34d9d in poll_iteration (ctx=0xbfb56b14) at transport.c:260
#11 0x0804a29b in main (argc=5, argv=0xbfb56bf4) at glusterfs.c:348
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
--
Amar Tumballi
http://amar.80x25.org
[bulde on #gluster/irc.gnu.org]
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
--
Anand V. Avati
--
Amar Tumballi
http://amar.80x25.org
[bulde on #gluster/irc.gnu.org]
--
Anand V. Avati
--
Anand V. Avati
--
Anand V. Avati
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
