This is complete one after restart of heketi pod and process log.
BR
Salam
From: "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
To: "Shaik Salam" <shaik.salam@xxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/24/2019 01:55 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
the logs you provided is not complete, not able to figure out which command is struck, can you reattach the complete output of `ps aux` and also attach complete heketi logs.
On Thu, Jan 24, 2019 at 1:41 PM Shaik Salam <shaik.salam@xxxxxxx> wrote:
Hi Madhu,
Please find requested info.
BR
Salam
From: Madhu Rajanna <mrajanna@xxxxxxxxxx>
To: Shaik Salam <shaik.salam@xxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, Michael Adam <madam@xxxxxxxxxx>
Date: 01/24/2019 01:33 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
the heketi logs you have attached is not complete i believe, can you povide the complete heketi logs
and also an we get the output of "ps aux" from the gluster pods ? I want to see if any lvm commands or gluster commands are "stuck".
On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <shaik.salam@xxxxxxx> wrote:
Hi Madhu.
I tried lot of times restarted heketi pod but not resolved.
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 0
New: 0
Stale: 0
Now you can see all operations are zero. Now I try to create single volume below is observation in-flight reaching slowly to 8.
sh-4.4# heketi-cli server operations infoCLI_SERVER=http://localhost:8080 ; export HEKETI_CLI_USE Operation Counts:
Total: 0
In-Flight: 6
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
sh-4.4# heketi-cli server operations info
Operation Counts:
Total: 0
In-Flight: 7
New: 0
Stale: 0
[negroni] Completed 200 OK in 186.286µs
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 166.294µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 186.411µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 179.796µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 131.108µs
[negroni] Started POST /volumes
[negroni] Started GET /operations
[negroni] Completed 200 OK in 111.392µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 265.023µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 179.364µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 295.058µs
[negroni] Started GET /operations
[negroni] Completed 200 OK in 146.857µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 403.166µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 193.554µs
But for pod volume is not creating.
| 1:15:36 PM |
Warning
| Provisioning failed | Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume Server busy. Retry operation later..
9 times in the last 2 minutes |
| 1:13:21 PM |
Warning
| Provisioning failed | Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume . 8 times in the last |
From: "Madhu Rajanna" <mrajanna@xxxxxxxxxx>
To: "Shaik Salam" <shaik.salam@xxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx List" <gluster-users@xxxxxxxxxxx>, "Michael Adam" <madam@xxxxxxxxxx>
Date: 01/24/2019 12:51 PM
Subject: Re: Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
"External email. Open with Caution"
HI Shaik,
can you provide me the outpout of $heketi-cli server operations info from heketi pod
as a workround you can try restarting the heketi pod. This will cause the current operations to go stale, but other pending pvcs may go to Bound state
Regards,
Madhu R
On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <shaik.salam@xxxxxxx>
wrote:
H Madhu,
Could you please have look my issue If you have time (atleast workaround).
I am unable to send mail to "John
Mulligan" <John_Mulligan@xxxxxxxxxx>"
who is currently handling issue
https://bugzilla.redhat.com/show_bug.cgi?id=1636912
BR
Salam
From: Shaik
Salam/HYD/TCS
To: "John
Mulligan" <John_Mulligan@xxxxxxxxxx>,
"Michael Adam" <madam@xxxxxxxxxx>,
"Madhu Rajanna" <mrajanna@xxxxxxxxxx>
Cc: "gluster-users@xxxxxxxxxxx
List" <gluster-users@xxxxxxxxxxx>
Date: 01/24/2019
12:21 PM
Subject: Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy
Hi All,
We are facing also following issue on openshift origin while we are creating pvc for pods. (atlease provide workaround to move further)
Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume
Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: create volume err: error creating volume Server busy. Retry operation later..
Please find heketidb dump and log
[negroni] Completed 429 Too Many Requests in 250.763µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 169.08µs
[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c
[negroni] Completed 404 Not Found in 148.125µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 496.624µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 101.673µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 209.681µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 103.595µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 297.594µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 96.75µs
[negroni] Started POST /volumes
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 477.007µs
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 165.38µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 488.253µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 171.836µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 208.59µs
[negroni] Started POST /volumes
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit (8)
[negroni] Completed 429 Too Many Requests in 125.141µs
[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606
[negroni] Completed 404 Not Found in 138.687µs
[negroni] Started POST /volumes
BR
Salam
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
--
Madhu Rajanna
Software Engineer
Red
Hat Bangalore, India
mrajanna@xxxxxxxxxx
M: +91-9741133155
--
Madhu Rajanna
Software Engineer
Red
Hat Bangalore, India
mrajanna@xxxxxxxxxx
M: +91-9741133155
--
Madhu Rajanna
Software Engineer
Red
Hat Bangalore, India
mrajanna@xxxxxxxxxx
M: +91-9741133155
Attachment:
heketi-pod-complete.log
Description: Binary data
[root@master ~]# oc get po NAME READY STATUS RESTARTS AGE glusterblock-storage-provisioner-dc-2-b46r0 1/1 Running 0 7d glusterfs-storage-31fjz 1/1 Running 0 104d glusterfs-storage-9h5cf 1/1 Running 0 104d glusterfs-storage-pnb67 1/1 Running 0 104d heketi-storage-6-4m61c 1/1 Running 0 2h [root@master ~]# oc rsh glusterfs-storage-31fjz sh-4.2# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 44276 4248 ? Ss 2018 19:00 /usr/sbin/init root 29 0.0 0.0 118040 59900 ? Ss 2018 3:06 /usr/lib/syste root 34 0.0 0.0 1706204 163716 ? SLsl 2018 77:07 /usr/sbin/dmev root 40 0.0 0.0 26096 1588 ? Ss 2018 0:15 /usr/sbin/cron root 63 0.0 0.0 389612 4116 ? Ssl 2018 0:00 /usr/bin/tcmu- root 64 0.0 0.0 94900 968 ? Ssl 2018 0:00 /usr/sbin/gssp rpc 65 0.0 0.0 69224 1048 ? Ss 2018 0:12 /sbin/rpcbind dbus 75 0.0 0.0 58068 2044 ? Ss 2018 0:13 /usr/bin/dbus- root 92 0.0 0.0 112816 2296 ? Ss 2018 0:00 /usr/sbin/sshd root 133 0.0 0.0 16544 2972 ? Ss 2018 1:43 /bin/bash /usr root 245 0.0 0.0 26380 1664 ? Ss 2018 0:24 /usr/lib/syste root 576 0.0 0.0 1423140 29968 ? Ssl 2018 35:47 /usr/sbin/glus root 869 0.0 0.0 1660308 44588 ? Ssl 2018 43:13 /usr/sbin/glus root 1934 0.0 0.0 2199692 98344 ? Ssl 2018 85:36 /usr/sbin/glus root 3907 0.0 0.0 2197092 48768 ? Ssl 2018 73:43 /usr/sbin/glus root 4486 0.0 0.0 2200992 56884 ? Ssl 2018 131:40 /usr/sbin/glus root 6510 20.2 0.3 27866580 925768 ? Ssl 2018 29368:27 /usr/sbin/gl root 11149 1.1 0.0 2265488 58244 ? Ssl 2018 1695:56 /usr/sbin/glu root 15336 0.0 0.0 11820 1788 pts/7 Ss+ Jan10 0:00 /bin/sh root 16908 0.0 0.0 1210152 18616 ? Ssl 2018 21:23 /usr/sbin/glus root 23994 0.0 0.0 11820 1732 pts/10 Ss+ Jan10 0:00 /bin/sh root 33780 0.3 0.0 574992 20348 ? Ssl Jan18 27:16 /usr/sbin/glus root 33821 0.0 0.0 202380 3752 ? Ssl Jan18 0:00 /usr/sbin/glus root 33863 0.0 0.0 2096820 59940 ? Ssl Jan18 0:35 /usr/sbin/glus root 42784 0.0 0.0 11820 1448 pts/12 Ss+ Jan17 0:00 /bin/sh root 54573 0.0 0.0 11820 1492 pts/6 Ss+ 2018 0:00 /bin/sh root 73119 0.0 0.0 15252 1988 pts/19 Ss+ 05:29 0:00 bash root 73126 0.0 0.0 1607008 27416 ? Ssl 2018 66:04 /usr/sbin/glus root 76135 0.0 0.0 1210152 18572 ? Ssl 2018 31:23 /usr/sbin/glus root 76350 0.0 0.0 11820 1648 pts/11 Ss+ 2018 0:00 /bin/sh root 87644 0.0 0.0 1473336 31268 ? Ssl 2018 29:10 /usr/sbin/glus root 90792 0.0 0.0 11820 1472 pts/13 Ss+ Jan22 0:00 /bin/sh root 91110 0.2 0.0 1671764 31624 ? Ssl 2018 355:21 /usr/sbin/glus root 92244 0.2 0.0 1539652 26904 ? Ssl 2018 393:21 /usr/sbin/glus root 95555 0.0 0.0 7768 352 ? S 09:48 0:00 sleep 120 root 95700 0.0 0.0 11820 1720 pts/14 Ss 09:50 0:00 /bin/sh root 95710 0.0 0.0 51712 1724 pts/14 R+ 09:50 0:00 ps aux root 103360 0.0 0.0 1510760 46724 ? Ssl 2018 26:44 /usr/sbin/glus root 109344 0.0 0.0 1935696 44764 ? Ssl 2018 121:18 /usr/sbin/glus root 109591 0.0 0.0 1933648 37484 ? Ssl 2018 72:42 /usr/sbin/glus root 109834 0.0 0.0 1802056 38212 ? Ssl 2018 83:08 /usr/sbin/glus root 121593 0.0 0.0 11820 1524 pts/0 Ss+ 2018 0:00 /bin/sh root 123388 0.0 0.0 1210152 20244 ? Ssl 2018 35:40 /usr/sbin/glus sh-4.2# exit exit [root@master ~]# oc rsh glusterfs-storage-9h5cf sh-4.2# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 43760 3948 ? Ss 2018 17:44 /usr/sbin/init root 29 0.0 0.0 85372 32084 ? Ss 2018 0:56 /usr/lib/syste root 30 0.0 0.0 1706204 162496 ? SLsl 2018 98:05 /usr/sbin/dmev root 41 0.0 0.0 26100 1596 ? Ss 2018 0:11 /usr/sbin/cron rpc 56 0.0 0.0 69224 1128 ? Ss 2018 0:10 /sbin/rpcbind root 63 0.0 0.0 94900 1232 ? Ssl 2018 0:00 /usr/sbin/gssp root 70 0.0 0.0 389612 7996 ? Ssl 2018 0:00 /usr/bin/tcmu- dbus 75 0.0 0.0 58068 2316 ? Ss 2018 0:10 /usr/bin/dbus- root 92 0.0 0.0 112816 4296 ? Ss 2018 0:00 /usr/sbin/sshd root 132 0.0 0.0 16540 2984 ? Ss 2018 1:44 /bin/bash /usr root 245 0.0 0.0 26456 1804 ? Ss 2018 0:21 /usr/lib/syste root 558 0.0 0.0 1493876 27780 ? Ssl 2018 20:22 /usr/sbin/glus root 901 0.0 0.0 1676948 51548 ? Ssl 2018 28:59 /usr/sbin/glus root 2130 0.0 0.0 2197092 44080 ? Ssl 2018 63:01 /usr/sbin/glus root 2713 0.0 0.0 2200732 55248 ? Ssl 2018 140:56 /usr/sbin/glus root 9064 1.1 0.0 2000744 60192 ? Ssl 2018 1684:57 /usr/sbin/glu root 11230 0.0 0.0 7768 348 ? S 09:50 0:00 sleep 120 root 11290 0.0 0.0 11820 1700 pts/6 Ss 09:50 0:00 /bin/sh root 11301 0.0 0.0 51712 1720 pts/6 R+ 09:50 0:00 ps aux root 23952 0.4 0.0 574992 13984 ? Ssl Jan18 39:22 /usr/sbin/glus root 23995 0.0 0.0 202380 5812 ? Ssl Jan18 0:00 /usr/sbin/glus root 24048 0.0 0.0 2096820 59372 ? Ssl Jan18 0:38 /usr/sbin/glus root 58228 0.0 0.0 11820 1692 pts/3 Ss+ 2018 0:00 /bin/sh root 69135 0.0 0.0 11820 1696 pts/4 Ss+ Jan18 0:00 /bin/sh root 70604 0.0 0.0 15252 2008 pts/1 Ss+ 2018 0:00 /bin/bash root 71770 0.0 0.0 1210152 20144 ? Ssl 2018 19:26 /usr/sbin/glus root 72776 0.0 0.0 1540692 32344 ? Ssl 2018 55:07 /usr/sbin/glus root 73279 0.0 0.0 15252 1976 pts/2 Ss+ 2018 0:00 /bin/bash root 82920 0.0 0.0 1407540 28548 ? Ssl 2018 19:30 /usr/sbin/glus root 86318 0.2 0.0 1737300 33392 ? Ssl 2018 375:05 /usr/sbin/glus root 87342 0.2 0.0 1539652 25968 ? Ssl 2018 345:32 /usr/sbin/glus root 97945 0.0 0.0 1469144 52052 ? Ssl 2018 20:24 /usr/sbin/glus root 98706 0.0 0.0 11820 1704 pts/5 Ss+ Jan20 0:00 /bin/sh root 103617 0.0 0.0 1867852 46592 ? Ssl 2018 82:57 /usr/sbin/glus root 103808 0.0 0.0 1933648 39320 ? Ssl 2018 77:33 /usr/sbin/glus root 103990 0.0 0.0 1802056 40996 ? Ssl 2018 88:53 /usr/sbin/glus root 105270 0.0 0.0 1210152 18388 ? Ssl 2018 18:13 /usr/sbin/glus root 120069 0.0 0.0 11820 1688 pts/0 Ss+ 2018 0:00 /bin/sh root 121771 0.0 0.0 1210152 21780 ? Ssl 2018 19:36 /usr/sbin/glus root 130924 0.0 0.0 2198912 95808 ? Ssl 2018 78:59 /usr/sbin/glus sh-4.2# exit exit [root@master ~]# oc rsh glusterfs-storage-pnb67 sh-4.2# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 44040 4012 ? Ss 2018 13:01 /usr/sbin/init root 29 0.0 0.0 72864 27152 ? Ss 2018 6:23 /usr/lib/syste root 33 0.0 0.0 1442828 160868 ? SLsl 2018 55:56 /usr/sbin/dmev root 48 0.0 0.0 26104 1600 ? Ss 2018 0:16 /usr/sbin/cron root 63 0.0 0.0 389612 6716 ? Ssl 2018 0:00 /usr/bin/tcmu- rpc 64 0.0 0.0 69224 1100 ? Ss 2018 0:13 /sbin/rpcbind root 65 0.0 0.0 94900 1092 ? Ssl 2018 0:00 /usr/sbin/gssp dbus 75 0.0 0.0 58068 2088 ? Ss 2018 0:56 /usr/bin/dbus- root 92 0.0 0.0 112816 3300 ? Ss 2018 0:00 /usr/sbin/sshd root 134 0.0 0.0 16540 2976 ? Ss 2018 1:41 /bin/bash /usr root 245 0.0 0.0 26488 1712 ? Ss 2018 0:33 /usr/lib/syste root 547 0.0 0.0 1357604 29068 ? Ssl 2018 25:01 /usr/sbin/glus root 861 0.0 0.0 1931520 51168 ? Ssl 2018 31:43 /usr/sbin/glus root 1980 0.0 0.0 2197092 45064 ? Ssl 2018 59:39 /usr/sbin/glus root 2552 0.0 0.0 2199432 52012 ? Ssl 2018 129:15 /usr/sbin/glus root 8809 1.0 0.0 2265228 59684 ? Ssl 2018 1517:25 /usr/sbin/glu root 24708 0.0 0.0 1210152 18020 ? Ssl 2018 21:52 /usr/sbin/glus root 47404 0.0 0.0 11820 1744 pts/11 Ss+ 08:08 0:00 /bin/sh root 52891 0.0 0.0 7768 344 ? S 09:49 0:00 sleep 120 root 52969 0.0 0.0 11820 1704 pts/16 Ss 09:51 0:00 /bin/sh root 52976 0.0 0.0 51712 1720 pts/16 R+ 09:51 0:00 ps aux root 70656 0.0 0.0 1210152 19764 ? Ssl 2018 22:57 /usr/sbin/glus root 73468 0.0 0.0 1540692 29904 ? Ssl 2018 54:23 /usr/sbin/glus root 75649 0.0 0.0 11820 1812 pts/6 Ss+ Jan21 0:00 /bin/sh root 81683 0.0 0.0 1407540 32992 ? Ssl 2018 23:41 /usr/sbin/glus root 85052 0.2 0.0 2066020 33628 ? Ssl 2018 358:41 /usr/sbin/glus root 86073 0.2 0.0 1605708 20324 ? Ssl 2018 300:41 /usr/sbin/glus root 97021 0.0 0.0 1570056 43760 ? Ssl 2018 24:24 /usr/sbin/glus root 98697 0.3 0.0 509456 13748 ? Ssl Jan21 15:04 /usr/sbin/glus root 98741 0.0 0.0 202380 1724 ? Ssl Jan21 0:00 /usr/sbin/glus root 98786 0.0 0.0 2031284 36664 ? Ssl Jan21 0:18 /usr/sbin/glus root 101995 0.0 0.0 11820 1704 pts/2 Ss+ Jan21 0:00 /bin/sh root 103063 0.0 0.0 1999444 41524 ? Ssl 2018 77:05 /usr/sbin/glus root 103276 0.0 0.0 1867852 48060 ? Ssl 2018 116:28 /usr/sbin/glus root 103457 0.0 0.0 1802056 46200 ? Ssl 2018 120:08 /usr/sbin/glus root 109041 0.0 0.0 15252 1972 pts/1 Ss+ 2018 0:00 /bin/bash root 120197 0.0 0.0 11820 1680 pts/3 Ss+ 2018 0:00 /bin/sh root 121840 0.0 0.0 1210152 21320 ? Ssl 2018 24:06 /usr/sbin/glus root 130810 0.0 0.0 2200732 92676 ? Ssl 2018 76:52 /usr/sbin/glus
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
