Could you please at least provide workaround for this issue.
We stuck from last two months and unable to use our setup
We tried following ways.
1. heketi pod restart
2. export pending operations and clear and import but still same issue when I try to create single volume at a time.
Please understand our concern and provide workaround.
BR
Salam
From: Shaik Salam/HYD/TCS
To: "John Mulligan" <jmulliga@xxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>, Madhu Rajanna <mrajanna@xxxxxxxxxx>
Date: 01/25/2019 04:03 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
Hi John,
Could you please have look my issue If you have time (atleast provide workaround).
Thanks in advance.
BR
Salam
From: "Shaik Salam" <shaik.salam@xxxxxxx>
To:
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/25/2019 02:55 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
Sent by: gluster-users-bounces@xxxxxxxxxxx
"External email. Open with Caution"
Hi John,
Please find db dump and heketi log. Here kernel version. Please let me know If you need more information.
[root@app2 ~]# uname -a
Linux app2.matrix.nokia.com 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Hardware: HP GEN8
OS; NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
From: "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
To: "Shaik Salam" <shaik.salam@xxxxxxx>, "John Mulligan" <jmulliga@xxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/24/2019 10:52 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
Adding John who is having more idea about how to debug this one.
@Shaik Salam can you some more info on the hardware on which you are running heketi (kernel details)
On Thu, Jan 24, 2019 at 7:42 PM Shaik Salam <shaik.salam@xxxxxxx> wrote:
Hi Madhu,
Sorry to disturb could you please provide atleast work around (to clear requests which stuck) to move further.
We are also not able to find root cause from glusterd logs. Please find attachment.
BR
Salam
From: Shaik Salam/HYD/TCS
To: "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/24/2019 04:12 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
Hi Madhu,
Please let me know If any other information required.
BR
Salam
From: Shaik Salam/HYD/TCS
To: "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/24/2019 03:23 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
Hi Madhu,
This is complete one after restart of heketi pod and process log.
BR
Salam
[attachment "heketi-pod-complete.log" deleted by Shaik Salam/HYD/TCS] [attachment "ps-aux.txt" deleted by Shaik Salam/HYD/TCS]
From: "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
To: "Shaik Salam" <shaik.salam@xxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/24/2019 01:55 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
the logs you provided is not complete, not able to figure out which command is struck, can you reattach the complete output of `ps aux` and also attach complete heketi logs.
On Thu, Jan 24, 2019 at 1:41 PM Shaik Salam <shaik.salam@xxxxxxx> wrote:
Hi Madhu,
Please find requested info.
BR
Salam
From: Madhu Rajanna <mrajanna@xxxxxxxxxx>
To: Shaik Salam <shaik.salam@xxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, Michael Adam <madam@xxxxxxxxxx>
Date: 01/24/2019 01:33 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
the heketi logs you have attached is not complete i believe, can you povide the complete heketi logs
and also an we get the output of "ps aux" from the gluster pods ? I want to see if any lvm commands or gluster commands are "stuck".
On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <shaik.salam@xxxxxxx> wrote:
Hi Madhu.
I tried lot of times restarted heketi pod but not resolved.
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 0
New: 0
Stale: 0
Now you can see all operations are zero. Now I try to create single volume below is observation in-flight reaching slowly to 8.
sh-4.4# heketi-cli server operations infoCLI_SERVER=http://localhost:8080 ; export HEKETI_CLI_USE Operation Counts:
Total: 0
In-Flight: 6
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
[negroni] Completed 200 OK in 186.286µs
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 166.294µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 186.411µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 179.796µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 131.108µs
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 111.392µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 265.023µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 179.364µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 295.058µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 146.857µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 403.166µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 193.554µs
But for pod volume is not creating.
| 1:15:36 PM |
Warning
| Provisioning failed | Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume Server busy. Retry operation later..
9 times in the last 2 minutes |
| 1:13:21 PM |
Warning
| Provisioning failed | Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume . 8 times in the last |
From: "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
To: "Shaik Salam" <shaik.salam@xxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/24/2019 12:51 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
HI Shaik,
can you provide me the outpout of $heketi-cli server operations info from heketi pod
as a workround you can try restarting the heketi pod. This will cause the current operations to go stale, but other pending pvcs may go to Bound state
Regards,
Madhu R
On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <shaik.salam@xxxxxxx>
wrote:
H Madhu,
Could you please have look my issue If you have time (atleast workaround).
I am unable to send mail to "John
Mulligan" <John_Mulligan@xxxxxxxxxx>"
who is currently handling issue
https://bugzilla.redhat.com/show_bug.cgi?id=1636912
BR
Salam
From: Shaik
Salam/HYD/TCS
To: "John
Mulligan" <John_Mulligan@xxxxxxxxxx>,
"Michael Adam" <madam@xxxxxxxxxx>,
"Madhu Rajanna" <mrajanna@xxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx
List" <gluster-users@xxxxxxxxxxx>
Date: 01/24/2019
12:21 PM
Subject: Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy
Hi All,
We are facing also following issue on openshift origin while we are creating pvc for pods. (atlease provide workaround to move further)
Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later..
Please find heketidb dump and log
[negroni] Completed 429 Too Many Requests in 250.763µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 169.08µs
[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c
[negroni] Completed 404 Not Found in 148.125µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 496.624µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 101.673µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 209.681µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 103.595µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 297.594µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 96.75µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 477.007µs
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 165.38µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 488.253µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 171.836µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 208.59µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 125.141µs
[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606
[negroni] Completed 404 Not Found in 138.687µs
[negroni] Started POST /volumes
BR
Salam
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
--
Madhu Rajanna
Software Engineer
Red
Hat Bangalore, India
mrajanna@xxxxxxxxxx
M: +91-9741133155
--
Madhu Rajanna
Software Engineer
Red
Hat Bangalore, India
mrajanna@xxxxxxxxxx
M: +91-9741133155
--
Madhu Rajanna
Software Engineer
Red
Hat Bangalore, India
mrajanna@xxxxxxxxxx
M: +91-9741133155
--
Madhu Rajanna
Software Engineer
Red
Hat Bangalore, India
mrajanna@xxxxxxxxxx
M: +91-9741133155
[attachment
"heketi-complete.log.txt" deleted by Shaik Salam/HYD/TCS] [attachment
"heketi-gluster.db.txt" deleted by Shaik Salam/HYD/TCS] _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
