Re: neither CPU nor IO bound, but throttled performance

[Gunther Schadow <gunther@xxxxxxxxxxxxxxxxx>]

Thank you Magnus. 68% steal. Indeed. You probably hit the target. Yes.

That explains the discrepancy. I need to watch and understand that CPU credits issue.

regards,
-Gunther

On 2/21/2019 4:08, Magnus Hagander wrote:

On Thu, Feb 21, 2019 at 12:34 AM Gunther <raj@xxxxxxxx> wrote:

Hi, I have an Amazon Linux based Postgresql 11 server here on a t2.medium EC2 instance.

It is serving 24 worker processes that read jobs from a queue (thanks to SELECT ... FOR UPDATE SKIP LOCKED!) and do jobs some of which are reading and writing business data to the database, others are only reading, and some don't hit the business data at all, only the queue.

Everything flows quite nicely. Except, I don't understand why I can't max out the CPU or the IO, instead, IO is almost negligible yet the CPU is at 30% hardly hitting 50%.

Here I give you a view of top:

top - 23:17:09 up 45 days,  2:07,  4 users,  load average: 20.32, 18.92, 13.80
Tasks: 338 total,  24 running, 111 sleeping,   0 stopped,   0 zombie
%Cpu(s): 28.7 us,  2.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 68.7 st

If I read that right, it's about 70% "steal". The description for this is "the percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.".

So basically, the CPU is spent dealing with other peoples VMs on the same hardware. Welcome to the multi-tenant cloud. 

In particular, I believe T series instances get a limited number of CPU "credits" per hours. My guess is you've hit this limit and are thus being throttled. T series are not intended for persistent workloads. Either way, this is probably a question better asked at the Amazon EC2 forums rather than PostgreSQL as you'll find more people who know the EC2 interactions there.

 

--

 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/