Hi,
I am keeping Raghvendra in loop and hope he can comment on the "Read being scheduled as slow fop".
Other than that, I would request you to provide following information to debug this issue.
1 - Profile information of the volume. You can find the steps here -
2 - Gluster volume <volname> status
3 - Gluster volume heal <volname> info
4 - Check the size of each bricks to see if it has been filled.
5 - gluster version of client and server.
6 - Host OS version of client and server.
---
Ashish
From: adi@xxxxxxxxxxxxxxxxxx
To: gluster-users@xxxxxxxxxxx
Sent: Thursday, October 10, 2019 8:08:24 PM
Subject: [Gluster-users] Gluster slow reads
Hi,
I have a very small distributed dispersed volume with 4+2 and 2 servers
with 3 bricks on each of them. The volume is mounted via fuse client on
another Linux server.
The volume worked well for a few months in this setup. However in the
last few days I have a very slow read speed (gluster uploads to the
gluster client via the mount point). By slow I mean 3-4 MB/sec on a
gigabit link. I don't have small files stored on gluster, the smallest
file is around 30-40 MB. The networking between the client and the
bricks is fine (all of them are connected in the same switch, no errors,
some iptraf tests directly between client and gluster servers are
looking good). On the same client I have mounted another 2 gluster
volumes from other servers, both of them are ok.
My 'gluster volume info' details are the following :
Volume Name: gluster4-vol
Type: Disperse
Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: gluster4:/export/sdb1/brick
Brick2: gluster5:/export/sdb1/brick
Brick3: gluster4:/export/sdc1/brick
Brick4: gluster5:/export/sdc1/brick
Brick5: gluster4:/export/sdd1/brick
Brick6: gluster5:/export/sdd1/brick
Options Reconfigured:
network.ping-timeout: 60
performance.client-io-threads: on
performance.io-thread-count: 32
cluster.readdir-optimize: on
performance.cache-size: 1GB
client.event-threads: 10
server.event-threads: 10
cluster.lookup-optimize: on
server.allow-insecure: on
storage.reserve: 0
transport.address-family: inet
nfs.disable: on
In the gluster client debug logs I noticed the following lines :
[2019-10-10 13:49:58.571879] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN
scheduled as fast fop
[2019-10-10 13:49:58.571964] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT
scheduled as fast fop
[2019-10-10 13:49:58.572058] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH
scheduled as normal fop
[2019-10-10 13:49:58.572728] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN
scheduled as fast fop
[2019-10-10 13:49:58.576275] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT
scheduled as fast fop
[2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:356:iot_schedule]
0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285
times between [2019-10-10 13:49:58.357922] and [2019-10-10
13:50:07.837047]
[2019-10-10 13:50:07.837068] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH
scheduled as normal fop
[2019-10-10 13:50:07.837165] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ
scheduled as slow fop
What does it mean exactly "READ scheduled as slow fop" ? Can I schedule
READs as normal or fast fop like the other operations ?
I'm using this gluster volume only for reading so I don't care about
writes right now.
Thanks.
________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
To: gluster-users@xxxxxxxxxxx
Sent: Thursday, October 10, 2019 8:08:24 PM
Subject: [Gluster-users] Gluster slow reads
Hi,
with 3 bricks on each of them. The volume is mounted via fuse client on
another Linux server.
last few days I have a very slow read speed (gluster uploads to the
gluster client via the mount point). By slow I mean 3-4 MB/sec on a
gigabit link. I don't have small files stored on gluster, the smallest
file is around 30-40 MB. The networking between the client and the
bricks is fine (all of them are connected in the same switch, no errors,
some iptraf tests directly between client and gluster servers are
looking good). On the same client I have mounted another 2 gluster
volumes from other servers, both of them are ok.
Type: Disperse
Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: gluster4:/export/sdb1/brick
Brick2: gluster5:/export/sdb1/brick
Brick3: gluster4:/export/sdc1/brick
Brick4: gluster5:/export/sdc1/brick
Brick5: gluster4:/export/sdd1/brick
Brick6: gluster5:/export/sdd1/brick
Options Reconfigured:
network.ping-timeout: 60
performance.client-io-threads: on
performance.io-thread-count: 32
cluster.readdir-optimize: on
performance.cache-size: 1GB
client.event-threads: 10
server.event-threads: 10
cluster.lookup-optimize: on
server.allow-insecure: on
storage.reserve: 0
transport.address-family: inet
nfs.disable: on
In the gluster client debug logs I noticed the following lines :
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN
scheduled as fast fop
[2019-10-10 13:49:58.571964] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT
scheduled as fast fop
[2019-10-10 13:49:58.572058] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH
scheduled as normal fop
[2019-10-10 13:49:58.572728] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN
scheduled as fast fop
[2019-10-10 13:49:58.576275] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT
scheduled as fast fop
[2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:356:iot_schedule]
0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285
times between [2019-10-10 13:49:58.357922] and [2019-10-10
13:50:07.837047]
[2019-10-10 13:50:07.837068] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH
scheduled as normal fop
[2019-10-10 13:50:07.837165] D [MSGID: 0]
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ
scheduled as slow fop
What does it mean exactly "READ scheduled as slow fop" ? Can I schedule
READs as normal or fast fop like the other operations ?
writes right now.
Thanks.
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/118564314 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/118564314 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
