Re: File system very slow

[vadud3@xxxxxxxxx]

- # gluster --version
glusterfs 7.5

- # gluster volume status atlassian 

Status of volume: atlassian
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick node1:/data/atlassian/gluster  49152     0          Y       1791
Brick node2:/data/atlassian/gluster  49152     0          Y       1773
Self-heal Daemon on localhost               N/A       N/A        Y       1807
Self-heal Daemon on node1.example.c
example.net                               N/A       N/A        Y       1778

Task Status of Volume atlassian
------------------------------------------------------------------------------
There are no active volume tasks

- # attached pre-du and during-du log from server

- I do not have a remote client. when I tried to run these

gluster volume profile your-volume start says already started since I am on the server

# setfattr -n trusted.io-stats-dump -v /tmp/io-stats-pre.txt /mnt runs but no output in /tmp/io-stats-pre.txt

- # gluster volume heal atlassian info 

Brick node1:/data/atlassian/gluster
Status: Connected
Number of entries: 0

Brick node2:/data/atlassian/gluster
Status: Connected
Number of entries: 0

Let me know if you need anything else. Appreciate your help

On Wed, May 27, 2020 at 3:27 AM Karthik Subrahmanya <ksubrahm@xxxxxxxxxx> wrote:

Hi,

Please provide the following information to understand the setup and debug this further:

- Which version of gluster you are using?

- 'gluster volume status atlassian' to confirm both bricks and shds are up or not

- Complete output of 'gluster volume profile atlassian info' before running 'du' and during 'du'. Redirect this output to separate files and attach them here

- Get the client side profile as well by following https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/

- 'gluster volume heal atlassian info' to check whether there are any pending heals and client side heal is contributing to this

Regards,

Karthik

On Wed, May 27, 2020 at 1:06 AM <vadud3@xxxxxxxxx> wrote:

I had a parsing error. It is Volume Name: atlassian

On Tue, May 26, 2020 at 3:12 PM <vadud3@xxxxxxxxx> wrote:

# gluster volume info

Volume Name: myvol
Type: Replicate
Volume ID: cbdef65c-79ea-496e-b777-b6a2981b29cf
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: node1:/data/foo/gluster
Brick2: node2:/data/foo/gluster
Options Reconfigured:
client.event-threads: 4
server.event-threads: 4
performance.stat-prefetch: on
network.inode-lru-limit: 16384
performance.md-cache-timeout: 1
performance.cache-invalidation: false
performance.cache-samba-metadata: false
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
performance.io-thread-count: 16
performance.cache-refresh-timeout: 5
performance.write-behind-window-size: 5MB
performance.cache-size: 1GB
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on

On Tue, May 26, 2020 at 3:06 PM Sunil Kumar Heggodu Gopala Acharya <sheggodu@xxxxxxxxxx> wrote:

Hi,

Please share the gluster volume information.

# gluster vol info

Regards,

Sunil kumar Acharya

On Wed, May 27, 2020 at 12:30 AM <vadud3@xxxxxxxxx> wrote:

I made the following changes for small file performance as suggested by http://blog.gluster.org/gluster-tiering-and-small-file-performance/

I am still seeing du -sh /data/shared taking 39 minutes.

Any other tuning I can do. Most of my files are 15K. Here is sample of small files with size and number of occurrences

FileSize.    # of occurrence

====        ============

1.1K 1122
1.1M 1040
1.2K 1281
1.2M 1357
1.3K 1149
1.3M 1098
1.4K 1119
1.5K 1189
1.6K 1036
1.7K 1169
11K 2157
12K 2398
13K 2402
14K 2406
15K 2426
16K 2386
17K 1986
18K 2037
19K 1829
2.0K 1027
2.1K 1048
2.4K 1013
20K 1585
21K 1713
22K 1590
23K 1371
24K 1428
25K 1444
26K 1391
27K 1217
28K 1485
29K 1282
30K 1303
31K 1275
32K 1296
33K 1058
36K 1023
37K 1107
39K 1092
41K 1034
42K 1187
46K 1030

On Mon, May 25, 2020 at 5:30 PM <vadud3@xxxxxxxxx> wrote:

time du -sh /data/shared

431G    /data/shared

real    45m49.992s
user    0m20.043s
sys    2m32.456s

gluster fs is extremely slow

Any suggestions on what settings to change to improve it?

--

Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

--

Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

--

Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

--

Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

--

Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Attachment: gvp.pre-du.log
Description: Binary data

Attachment: gvp-during-du.log
Description: Binary data

________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users