Re: High load on glusterfs client

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



When it happens again, I’ll check the output. Thanks!

 

Von: Krutika Dhananjay [mailto:kdhananj@xxxxxxxxxx]
Gesendet: Dienstag, 15. März 2016 04:28
An: Gumprich, Sebastian
Cc: gluster-user
Betreff: Re: [Gluster-users] High load on glusterfs client

 

Looks like a case of gfid split-brain.

What does the output of `gluster volume heal <VOL> info split-brain` say?

-Krutika

 

On Mon, Mar 14, 2016 at 1:05 PM, <Sebastian.Gumprich@xxxxxxxxxxxxx> wrote:

Today, after setting the self-heal options to off, the problem happened again.

 

Here’s what happens when I try to heal the problems:

 

[root@nfs02 gluster volume heal opt info

Brick nfs01:/opt/bkk

Number of entries: 0

 

Brick nfs02:/opt/bkk

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Standalone_template_source_948126c20bcb68e1839bd1cb11b2326776a5a89a.php

<gfid:869bab4e-f83f-4876-b619-a7c008361bd2>

<gfid:e5cd468a-c3cc-453f-bd10-67d4474bd57c>

<gfid:e792eef8-a58f-4808-985b-dc906a707593>

<gfid:7ab28ace-94ae-468e-9a11-0b5a3a42a9d7>

<gfid:ae890cd6-97b8-493a-a6ea-bedf45707f17>

<gfid:7bd13cdb-4c6c-4aa5-a7df-1f3cf78bfca3>

<gfid:d7f63c6b-0b69-4271-827e-16bb1bfa689e>

<gfid:a128f009-1edc-47fb-8e53-9490887f9937>

<gfid:d0a0d264-ebc2-4370-8114-4732fb894e66>

<gfid:317f6a2e-f243-4ead-a4f4-8508b70d76d2>

<gfid:082a3550-15ad-4719-a550-c60c7f8c8791>

<gfid:e5d7c0fb-42e5-4f81-aebd-9f55fb12d8cf>

<gfid:4ec4830b-c65e-451b-a754-d48bd4c385a6>

<gfid:f0453756-50b4-423f-84b5-82bf840ae465>

<gfid:264d0bba-7208-4866-9820-e561439afbb8>

<gfid:ebe43f9f-6b55-40bb-8a59-750ab3dc7b1f>

<gfid:6f235178-5981-42d8-83fe-ce222c22600a>

<gfid:c2ee0feb-8edc-4fe3-bda6-55845c6ab351>

/releases/1.0.1/typo3temp/locks/b3e7327e62654b234caf95033dd5f5db

<gfid:f450c66b-d6ae-4c65-a95a-f13ef5285900>

<gfid:98aabb13-dafe-474c-87e0-2741645a1e6b>

<gfid:1bebfba1-13c1-4c22-8b1b-3d34e5d7b968>

<gfid:97e5a8cc-7c66-47b0-8dbd-9f9098b419d9>

<gfid:438a2995-32d2-4a21-ae5d-45593a74addf>

<gfid:008f5f63-9dda-408b-abf7-2201b029462e>

<gfid:f89a5507-f9c7-45f1-8760-ea9a897c4659>

<gfid:3da89478-9c08-4c57-a44f-0df2dc59efb4>

<gfid:f0969ff6-f49a-4497-a24f-648d4e130775>

<gfid:6c7f9d1e-64e5-43f7-a454-9f823641f0ae>

<gfid:1d7c6cc0-fd07-43bb-8caf-30e80544e516>

/releases/1.0.1/typo3temp/locks/dabccbb64a6426cf9b1005b79f5765a9

Number of entries: 32

 

[root@nfs02 ~]# gluster volume heal opt full

Launching heal operation to perform full self heal on volume opt has been unsuccessful

 

[root@nfs02 ~]# gluster volume heal opt

Launching heal operation to perform index self heal on volume opt has been successful

Use heal info commands to check status

 

[root@nfs02 ~]# gluster volume heal opt info

Brick nfs01:/opt/bkk

Number of entries: 0

 

Brick nfs02:/opt/bkk

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Standalone_template_source_948126c20bcb68e1839bd1cb11b2326776a5a89a.php

<gfid:869bab4e-f83f-4876-b619-a7c008361bd2>

<gfid:e5cd468a-c3cc-453f-bd10-67d4474bd57c>

<gfid:e792eef8-a58f-4808-985b-dc906a707593>

<gfid:7ab28ace-94ae-468e-9a11-0b5a3a42a9d7>

<gfid:ae890cd6-97b8-493a-a6ea-bedf45707f17>

<gfid:7bd13cdb-4c6c-4aa5-a7df-1f3cf78bfca3>

<gfid:d7f63c6b-0b69-4271-827e-16bb1bfa689e>

<gfid:a128f009-1edc-47fb-8e53-9490887f9937>

<gfid:d0a0d264-ebc2-4370-8114-4732fb894e66>

<gfid:317f6a2e-f243-4ead-a4f4-8508b70d76d2>

<gfid:082a3550-15ad-4719-a550-c60c7f8c8791>

<gfid:e5d7c0fb-42e5-4f81-aebd-9f55fb12d8cf>

<gfid:4ec4830b-c65e-451b-a754-d48bd4c385a6>

<gfid:f0453756-50b4-423f-84b5-82bf840ae465>

<gfid:264d0bba-7208-4866-9820-e561439afbb8>

<gfid:ebe43f9f-6b55-40bb-8a59-750ab3dc7b1f>

<gfid:6f235178-5981-42d8-83fe-ce222c22600a>

<gfid:c2ee0feb-8edc-4fe3-bda6-55845c6ab351>

/releases/1.0.1/typo3temp/locks/b3e7327e62654b234caf95033dd5f5db

<gfid:f450c66b-d6ae-4c65-a95a-f13ef5285900>

<gfid:98aabb13-dafe-474c-87e0-2741645a1e6b>

<gfid:1bebfba1-13c1-4c22-8b1b-3d34e5d7b968>

<gfid:97e5a8cc-7c66-47b0-8dbd-9f9098b419d9>

<gfid:438a2995-32d2-4a21-ae5d-45593a74addf>

<gfid:008f5f63-9dda-408b-abf7-2201b029462e>

<gfid:f89a5507-f9c7-45f1-8760-ea9a897c4659>

<gfid:3da89478-9c08-4c57-a44f-0df2dc59efb4>

<gfid:f0969ff6-f49a-4497-a24f-648d4e130775>

<gfid:6c7f9d1e-64e5-43f7-a454-9f823641f0ae>

<gfid:1d7c6cc0-fd07-43bb-8caf-30e80544e516>

Number of entries: 31

 

 

Here’s the log to it:

[2016-03-14 07:25:34.593287] W [MSGID: 108008] [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check] 0-opt-replicate-0: GFID mismatch for <gfid:7bfa065f-50c7-43d6-b513-df6cd76fa144>/Standalone_layout_Mail_fa22189c871e1e7c1a36b59209ef0812d5e709e9.php 082a3550-15ad-4719-a550-c60c7f8c8791 on opt-client-1 and c2f289ca-6058-46ae-af8a-18491975eb7d on opt-client-0

 

 

[root@-nfs02 ~]# gluster volume info

 

Volume Name: opt

Type: Replicate

Volume ID: 5b77070f-5378-45ec-9eda-5f7dd007ff8a

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: nfs01:/opt/bkk

Brick2: nfs02:/opt/bkk

Options Reconfigured:

cluster.metadata-self-heal: off

cluster.data-self-heal: off

cluster.entry-self-heal: off

performance.md-cache-timeout: 1

performance.cache-max-file-size: 2MB

performance.io-thread-count: 16

network.ping-timeout: 42

performance.write-behind-window-size: 4MB

performance.read-ahead: off

performance.cache-refresh-timeout: 10

performance.cache-size: 512MB

performance.quick-read: off

performance.readdir-ahead: on

 

 

Any other ideas?

 

Thanks!

 

Von: Krutika Dhananjay [mailto:kdhananj@xxxxxxxxxx]
Gesendet: Dienstag, 8. März 2016 04:04
An: Gumprich, Sebastian
Cc: gluster-users@xxxxxxxxxxx
Betreff: Re: [Gluster-users] High load on glusterfs client

 

Could you try disabling client-side heals and see if it works for you?

Here's what you'd need to do:

#gluster volume set <VOL> entry-self-heal off

#gluster volume set <VOL> data-self-heal off

#gluster volume set <VOL> metadata-self-heal off

-Krutika

 

On Wed, Mar 2, 2016 at 12:37 AM, <Sebastian.Gumprich@xxxxxxxxxxxxx> wrote:

Hello everyone,

 

I’m experiencing high load on our glusterfs clients.

 

Here’s the setup:

 

There are to glusterfs server:

 

Nfs01 and nfs02 with the following configuration:

 

[root nfs01 ~]# gluster volume info opt

 

Volume Name: opt

Type: Replicate

Volume ID: 5b77070f-5378-45ec-9eda-5f7dd007ff8a

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1:  nfs01:/opt/bkk

Brick2: nfs02:/opt/bkk

Options Reconfigured:

performance.readdir-ahead: on

performance.quick-read: off

performance.cache-size: 512MB

performance.cache-refresh-timeout: 10

performance.read-ahead: off

performance.write-behind-window-size: 4MB

network.ping-timeout: 2

performance.io-thread-count: 16

performance.cache-max-file-size: 2MB

performance.md-cache-timeout: 1

 

Then there are two clients (web01 and web02) that mount the brick via a virtual ip-address (nfs-VIP):

nfs-VIP:/opt on /opt/bkk type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

 

operating system on all server is CentOS Linux release 7.2.1511 (Core).

Glusterfs version is glusterfs 3.7.6 built on Nov  9 2015 15:20:26

 

On the brick lies the PHP dynamic webcontent from a typo3 CMS.

 

On the client (web01) the following is logged in the gluster.log:

 

iner_08850598886fb5f39c9cf1d269d7e20677f97ede.php>, e09948dd-1e9b-4430-8f55-3df64cda2385 on opt-client-1 and ba80a475-7b83-4c83-bd0c-798a108bfb63 on opt-client-0. Skipping conservative merge on the file.

[2016-03-01 18:40:50.570040] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-opt-replicate-0: Gfid mismatch detected for <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_layout_Detail_html_bd113d9c433c8f88376e47547db3b94e698a5ecd.php>, 739ee14c-2d5d-458b-bffd-83595bfcbe6a on opt-client-1 and 5a311733-731e-4478-ad3c-a70fbf66ba30 on opt-client-0. Skipping conservative merge on the file.

[2016-03-01 18:40:50.572992] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-opt-replicate-0: Gfid mismatch detected for <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_partial_Detail_FalMediaContainer_9c1b3fd40fca9019726b3f6b8bc04618ffadab7b.php>, bb6907a1-ce80-4e03-92df-6fbc69d24a4d on opt-client-1 and 6f88aa67-cb81-4c26-94b2-e3aaa8704e8d on opt-client-0. Skipping conservative merge on the file.

[2016-03-01 18:40:50.791704] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-opt-replicate-0: Gfid mismatch detected for <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_action_create_f40464a6a7f73d86cda514065167d59a7ddece73.php>, 5e5b224b-ea20-4d38-8504-61b24f5d6a3b on opt-client-1 and fab07af5-2aa5-4873-a6e2-6265ec78e304 on opt-client-0. Skipping conservative merge on the file.

[2016-03-01 18:40:54.085964] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-opt-replicate-0: Gfid mismatch detected for <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/News_News_action_detail_8d30b654cd8343fe40616b8a2f8a5343b1ed776e.php>, 4d75a687-b9ab-4f97-b698-38668d1981ae on opt-client-1 and 110b315e-2e28-4859-a8b9-e0f1629faa3c on opt-client-0. Skipping conservative merge on the file.

[2016-03-01 18:40:56.153651] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-opt-replicate-0: Gfid mismatch detected for <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_layout_Default_aae217b167ad82f4b1258bb01fa73f305844dbd8.php>, 6f7e2709-8c14-486a-85a2-a3cb48af4ca5 on opt-client-1 and 6ab62408-0406-4834-96b9-a51e18441d4c on opt-client-0. Skipping conservative merge on the file.

[2016-03-01 18:41:05.476126] I [MSGID: 108026] [afr-self-heal-entry.c:593:afr_selfheal_entry_do] 0-opt-replicate-0: performing entry selfheal on 7a922c37-48d0-4dfb-8abb-18a435c948af

[2016-03-01 18:41:05.597093] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-opt-replicate-0: Completed entry selfheal on 7a922c37-48d0-4dfb-8abb-18a435c948af. source=1 sinks=0

[2016-03-01 18:41:05.790944] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-opt-replicate-0: Gfid mismatch detected for <4c6dda77-6a2b-4996-bca4-9ace4cee45cc/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php>, 118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0. Skipping conservative merge on the file.

[2016-03-01 18:41:06.649695] W [MSGID: 108008] [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check] 0-opt-replicate-0: GFID mismatch for <gfid:4c6dda77-6a2b-4996-bca4-9ace4cee45cc>/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php 118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0

[2016-03-01 18:41:06.661277] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 184415191: LOOKUP() /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php => -1 (Input/output error)

[2016-03-01 18:41:06.680968] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 184422672: LOOKUP() /releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php => -1 (Input/output error)

[2016-03-01 18:41:06.680222] W [MSGID: 108008] [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check] 0-opt-replicate-0: GFID mismatch for <gfid:4c6dda77-6a2b-4996-bca4-9ace4cee45cc>/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php 118668d9-608a-477a-b655-bcc6c2298bf4 on opt-client-1 and a87943a4-e18a-4642-adff-1ad765496533 on opt-client-0

 

There are many more of these entries, this is just a really small excerpt. The files that have a mismatch are tempary php-cache files.

When I delete these files, the load goes down and the files in the volume heal info become less (see below).

 

Here’s the output of gluster volume heal opt info. Note that this output is *after* deleting most of the cache files, before that there were many more entries.

 

[root@nfs01 fluid_template]# gluster volume heal opt info

Brick nfs01:/opt/bkk

<gfid:23fc1027-0aec-4b84-9ffb-c164a9d43d20>

<gfid:92cb9dde-2721-4c11-93a6-2582ed9edd5d>

<gfid:a0dbcf8a-67f8-4870-ab57-3d5d1218601c>

<gfid:947cbcc4-1978-4b9e-b726-2acd0a4fda5a>

<gfid:440b9b36-bad5-4cb6-b935-8a004132340a>

Number of entries: 5

 

Brick nfs02:/opt/bkk

/releases/1.0.1/typo3temp/Cache/Code/fluid_template - Possibly undergoing heal

 

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Select_7bf809152d985037de761d8d375d286e44b4f13a.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_GoogleAdwordsConversion_f7254aeb252ea43cd89f9051b5a43109d47938f1.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_File_4b3a3f667c475577847aa77118f3af5666ecb2c6.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_form_f0755f8526150f023fd98252b510a40c49586dbd.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Input_b3e08744b23680f0a14e60e716f9994d9580e3f4.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_action_detail_8d30b654cd8343fe40616b8a2f8a5343b1ed776e.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_layout_Detail_html_bd113d9c433c8f88376e47547db3b94e698a5ecd.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_Opengraph_b98680f3686dccf00e22181e66d11ca9de7a44bd.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_FalMediaContainer_9c1b3fd40fca9019726b3f6b8bc04618ffadab7b.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/News_News_partial_Detail_MediaContainer_08850598886fb5f39c9cf1d269d7e20677f97ede.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Text_c10766db8d335d5cd9555878aef5d886dcb6926e.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_action_create_f40464a6a7f73d86cda514065167d59a7ddece73.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_layout_Default_aae217b167ad82f4b1258bb01fa73f305844dbd8.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_HoneyPod_fc83c414f744612c3cb44c8827372a30f17791d0.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Textarea_39d24d8e3e2813636dfff2a89b7cefb8e9117c97.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Submit_86e69c50ccebf20584db2e3c74859373c53d320f.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Check_a2a11c64ac58dab16eab29e4cda88518c15a4d25.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Misc_FormError_7cade8e8fc1d23c761360c0efbe8cb145eed2e39.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Radio_d038a263b5ea81f0f7795e1e47516e5e2937cbd9.php

/releases/1.0.1/typo3temp/Cache/Code/fluid_template/Powermail_Form_partial_Form_Hidden_a7651f5498e0d36b4e2eae5fca015ef0e9365067.php

Number of entries: 21

 

Here are some heal-infos during the high load:

Starting time of crawl: Tue Mar  1 19:09:45 2016

 

Ending time of crawl: Tue Mar  1 19:09:52 2016

 

Type of crawl: INDEX

No. of entries healed: 2

No. of entries in split-brain: 0

No. of heal failed entries: 168

 

 

And here’s the performance monitoring info during 60 seconds of high load:

 

Brick: nfs01:/opt/bkk

------------------------------

Cumulative Stats:

   Block Size:                  1b+                   2b+                   4b+

No. of Reads:                   18                    33                   308

No. of Writes:                   64                    88                  2994

 

   Block Size:                  8b+                  16b+                  32b+

No. of Reads:                  117                   154                 15612

No. of Writes:                  370                   369                  1432

 

   Block Size:                 64b+                 128b+                 256b+

No. of Reads:                 3721                 12884                 19917

No. of Writes:                 7585                900221                135011

 

   Block Size:                512b+                1024b+                2048b+

No. of Reads:                23929                 12251                 19835

No. of Writes:                63067                 30950                 23540

 

   Block Size:               4096b+                8192b+               16384b+

No. of Reads:                 9096                  9449                  5566

No. of Writes:                40455                 36397                 13926

 

   Block Size:              32768b+               65536b+              131072b+

No. of Reads:                 5159                  6055                 34001

No. of Writes:                20722                  6600                 12762

 

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop

---------   -----------   -----------   -----------   ------------        ----

      0.00       0.00 us       0.00 us       0.00 us         212065      FORGET

      0.00       0.00 us       0.00 us       0.00 us        4118713     RELEASE

      0.00       0.00 us       0.00 us       0.00 us        9931097  RELEASEDIR

      0.00      47.00 us      47.00 us      47.00 us              1    GETXATTR

      0.00     124.00 us     124.00 us     124.00 us              1     XATTROP

      0.00     144.00 us     144.00 us     144.00 us              1      UNLINK

      0.00      37.17 us      35.00 us      41.00 us              6      STATFS

      0.01      36.84 us      32.00 us      61.00 us             19       FSTAT

      0.01      48.80 us      44.00 us      62.00 us             20        STAT

      0.05      75.52 us      54.00 us     156.00 us             48 REMOVEXATTR

      0.05      81.69 us      69.00 us     146.00 us             48     SETATTR

      0.05      37.16 us      14.00 us     436.00 us            109       FLUSH

      0.05      38.31 us      18.00 us     115.00 us            108    FINODELK

      0.06      73.16 us      46.00 us     170.00 us             63        OPEN

      0.10      38.70 us      20.00 us     167.00 us            195     INODELK

      0.11     335.37 us      40.00 us     563.00 us             27     READDIR

      0.15      50.58 us      27.00 us     392.00 us            232     OPENDIR

      0.15      81.74 us      35.00 us     215.00 us            144    FXATTROP

      0.16     130.53 us      68.00 us     697.00 us            100       WRITE

      0.93    1532.04 us     171.00 us   15356.00 us             48      CREATE

      1.40     284.90 us      25.00 us    1146.00 us            390    READDIRP

     10.20      52.29 us      28.00 us    2323.00 us          15482    READLINK

     18.94      33.25 us      11.00 us   27242.00 us          45219     ENTRYLK

     67.58      93.74 us      32.00 us   27521.00 us          57223      LOOKUP

 

    Duration: 6492593 seconds

   Data Read: 5679496942 bytes

Data Written: 4510536316 bytes

 

Interval 1 Stats:

   Block Size:               4096b+                8192b+               16384b+

No. of Reads:                    0                     0                     0

No. of Writes:                    1                    50                     2

 

   Block Size:              32768b+

No. of Reads:                    0

No. of Writes:                   47

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop

---------   -----------   -----------   -----------   ------------        ----

      0.00       0.00 us       0.00 us       0.00 us             48      FORGET

      0.00       0.00 us       0.00 us       0.00 us            104     RELEASE

      0.00       0.00 us       0.00 us       0.00 us            231  RELEASEDIR

      0.00     124.00 us     124.00 us     124.00 us              1     XATTROP

      0.00     144.00 us     144.00 us     144.00 us              1      UNLINK

      0.00      36.40 us      35.00 us      37.00 us              5      STATFS

      0.01      36.84 us      32.00 us      61.00 us             19       FSTAT

      0.01      48.80 us      44.00 us      62.00 us             20        STAT

      0.05      75.52 us      54.00 us     156.00 us             48 REMOVEXATTR

      0.05      37.69 us      14.00 us     436.00 us            101       FLUSH

      0.06      81.69 us      69.00 us     146.00 us             48     SETATTR

      0.06      74.26 us      46.00 us     170.00 us             53        OPEN

      0.06      38.31 us      18.00 us     115.00 us            108    FINODELK

      0.10     311.33 us      40.00 us     563.00 us             24     READDIR

      0.11      38.70 us      20.00 us     167.00 us            195     INODELK

      0.16      50.58 us      27.00 us     392.00 us            231     OPENDIR

      0.17      81.74 us      35.00 us     215.00 us            144    FXATTROP

      0.18     130.53 us      68.00 us     697.00 us            100       WRITE

      1.03    1532.04 us     171.00 us   15356.00 us             48      CREATE

      1.56     284.90 us      25.00 us    1146.00 us            390    READDIRP

     11.31      52.27 us      28.00 us    2323.00 us          15395    READLINK

     18.24      33.67 us      11.00 us   27242.00 us          38571     ENTRYLK

     66.83      94.49 us      32.00 us   27521.00 us          50338      LOOKUP

 

    Duration: 68 seconds

   Data Read: 0 bytes

Data Written: 3347998 bytes

 

Brick: nfs02:/opt/bkk

------------------------------

Cumulative Stats:

   Block Size:                  1b+                   2b+                   4b+

No. of Reads:                   26                    49                   541

No. of Writes:                   64                    94                  3848

 

   Block Size:                  8b+                  16b+                  32b+

No. of Reads:                  218                   205                  1267

No. of Writes:                  452                   417                  1448

 

   Block Size:                 64b+                 128b+                 256b+

No. of Reads:                 6097                 39042                 11111

No. of Writes:                 8617                924503                136768

 

   Block Size:                512b+                1024b+                2048b+

No. of Reads:               120819                 37802                 16506

No. of Writes:                64399                 35996                 24999

 

   Block Size:               4096b+                8192b+               16384b+

No. of Reads:                76162                 20449                 10948

No. of Writes:                41302                 37488                 14034

 

   Block Size:              32768b+               65536b+              131072b+

No. of Reads:                 7733                  7306                 31648

No. of Writes:                20849                  6750                 12886

 

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop

---------   -----------   -----------   -----------   ------------        ----

      0.00       0.00 us       0.00 us       0.00 us         231622      FORGET

      0.00       0.00 us       0.00 us       0.00 us        6123626     RELEASE

      0.00       0.00 us       0.00 us       0.00 us       10869781  RELEASEDIR

      0.00     116.00 us     116.00 us     116.00 us              1     XATTROP

      0.00      40.00 us      38.00 us      43.00 us              6      STATFS

      0.01      40.85 us      33.00 us      97.00 us             13       FSTAT

      0.01      46.26 us      29.00 us     100.00 us             23        STAT

      0.04      73.96 us      53.00 us     150.00 us             48 REMOVEXATTR

      0.04      76.85 us      61.00 us      99.00 us             48     SETATTR

      0.04      35.83 us      28.00 us     155.00 us            103      UNLINK

      0.04      36.28 us      15.00 us     142.00 us            112    FINODELK

      0.05      38.68 us      13.00 us     220.00 us            133       FLUSH

      0.09     324.11 us      28.00 us     589.00 us             28     READDIR

      0.12      85.88 us      36.00 us     215.00 us            144    FXATTROP

      0.12     124.55 us      76.00 us     192.00 us            100       WRITE

      0.13      78.76 us      19.00 us     529.00 us            161    GETXATTR

      0.20      78.65 us      43.00 us     384.00 us            261        OPEN

      0.23      54.09 us       2.00 us     260.00 us            426     OPENDIR

      0.60    1261.23 us     174.00 us   10655.00 us             48      CREATE

      0.66      81.15 us      17.00 us    9254.00 us            819     INODELK

      1.21     279.97 us      23.00 us    1587.00 us            434    READDIRP

      8.10      52.61 us      27.00 us    1283.00 us          15496    READLINK

     15.48      34.54 us      10.00 us   13810.00 us          45133     ENTRYLK

     72.84     104.30 us      14.00 us   14613.00 us          70322      LOOKUP

 

    Duration: 6492593 seconds

   Data Read: 6308054987 bytes

Data Written: 4579768980 bytes

 

Interval 1 Stats:

   Block Size:               4096b+                8192b+               16384b+

No. of Reads:                    0                     0                     0

No. of Writes:                    1                    50                     2

 

   Block Size:              32768b+

No. of Reads:                    0

No. of Writes:                   47

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop

---------   -----------   -----------   -----------   ------------        ----

      0.00       0.00 us       0.00 us       0.00 us             48      FORGET

      0.00       0.00 us       0.00 us       0.00 us            286     RELEASE

      0.00       0.00 us       0.00 us       0.00 us            400  RELEASEDIR

      0.00     116.00 us     116.00 us     116.00 us              1     XATTROP

      0.00      40.40 us      38.00 us      43.00 us              5      STATFS

      0.01      40.85 us      33.00 us      97.00 us             13       FSTAT

      0.01      46.14 us      29.00 us     100.00 us             22        STAT

      0.04      73.96 us      53.00 us     150.00 us             48 REMOVEXATTR

      0.04      76.85 us      61.00 us      99.00 us             48     SETATTR

      0.04      35.83 us      28.00 us     155.00 us            103      UNLINK

      0.05      36.28 us      15.00 us     142.00 us            112    FINODELK

      0.05      39.64 us      13.00 us     220.00 us            117       FLUSH

      0.10     330.28 us      28.00 us     589.00 us             25     READDIR

      0.14      85.88 us      36.00 us     215.00 us            144    FXATTROP

      0.14     124.55 us      76.00 us     192.00 us            100       WRITE

      0.14      79.13 us      19.00 us     529.00 us            159    GETXATTR

      0.22      79.51 us      43.00 us     384.00 us            235        OPEN

      0.25      54.27 us       2.00 us     260.00 us            400     OPENDIR

      0.70    1261.23 us     174.00 us   10655.00 us             48      CREATE

      0.77      81.15 us      17.00 us    9254.00 us            819     INODELK

      1.26     283.31 us      23.00 us    1587.00 us            386    READDIRP

      7.67      52.88 us      27.00 us    1283.00 us          12595    READLINK

     15.47      34.83 us      10.00 us   13810.00 us          38576     ENTRYLK

     72.91     105.05 us      14.00 us   14613.00 us          60295      LOOKUP

 

    Duration: 68 seconds

   Data Read: 0 bytes

Data Written: 3347998 bytes

 

 

 

Can anybody tell me how to fix the problem with the high load and these cache files?

 

Thanks in advance!

 

Regards

Sebastian


_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

 

 

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux