Gluster slow reads

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




 Hi,

I have a very small distributed dispersed volume with 4+2 and 2 servers with 3 bricks on each of them. The volume is mounted via fuse client on another Linux server.

The volume worked well for a few months in this setup. However in the last few days I have a very slow read speed (gluster uploads to the gluster client via the mount point). By slow I mean 3-4 MB/sec on a gigabit link. I don't have small files stored on gluster, the smallest file is around 30-40 MB. The networking between the client and the bricks is fine (all of them are connected in the same switch, no errors, some iptraf tests directly between client and gluster servers are looking good). On the same client I have mounted another 2 gluster volumes from other servers, both of them are ok.

 My 'gluster volume info' details are the following :

Volume Name: gluster4-vol
Type: Disperse
Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: gluster4:/export/sdb1/brick
Brick2: gluster5:/export/sdb1/brick
Brick3: gluster4:/export/sdc1/brick
Brick4: gluster5:/export/sdc1/brick
Brick5: gluster4:/export/sdd1/brick
Brick6: gluster5:/export/sdd1/brick
Options Reconfigured:
network.ping-timeout: 60
performance.client-io-threads: on
performance.io-thread-count: 32
cluster.readdir-optimize: on
performance.cache-size: 1GB
client.event-threads: 10
server.event-threads: 10
cluster.lookup-optimize: on
server.allow-insecure: on
storage.reserve: 0
transport.address-family: inet
nfs.disable: on


 In the gluster client debug logs I noticed the following lines :

[2019-10-10 13:49:58.571879] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN scheduled as fast fop [2019-10-10 13:49:58.571964] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT scheduled as fast fop [2019-10-10 13:49:58.572058] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH scheduled as normal fop [2019-10-10 13:49:58.572728] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN scheduled as fast fop [2019-10-10 13:49:58.576275] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT scheduled as fast fop [2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal] 0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to flush least recently used log message to disk The message "D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285 times between [2019-10-10 13:49:58.357922] and [2019-10-10 13:50:07.837047] [2019-10-10 13:50:07.837068] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH scheduled as normal fop [2019-10-10 13:50:07.837165] D [MSGID: 0] [io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ scheduled as slow fop


What does it mean exactly "READ scheduled as slow fop" ? Can I schedule READs as normal or fast fop like the other operations ?

I'm using this gluster volume only for reading so I don't care about writes right now.


 Thanks.

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux