Weird CPU utilization patterns with Postgres

[István <leccine@xxxxxxxxx>]

Hi,

We are having a really interesting problem with our Postgres 9.3 instance in our infrastructure.

Few days ago our box started to show huge CPU spikes while the IO Wait is negligible on the box. After a while I have installed perf and started to monitor the Postgres master process and here is what I have found:

Samples: 372K of event 'cycles', Event count (approx.): 110095222173, ThreaSamples: 372K of event 'cycles', Event count (approx.): 1100 93.65%  libc-2.12.so  [.] __strcoll_l
  0.97%  libc-2.12.so  [.] memcpy
  0.90%  postgres      [.] slot_getattr
  0.88%  postgres      [.] nocachegetattr
  0.64%  postgres      [.] varstr_cmp
  0.52%  libc-2.12.so  [.] __strcmp_sse42
  0.43%  postgres      [.] hash_any
  0.32%  postgres      [.] pg_detoast_datum_packed
  0.31%  libc-2.12.so  [.] __strlen_sse2
  0.22%  postgres      [.] bttextcmp
  0.18%  postgres      [.] ExecStoreTuple
  0.14%  postgres      [.] MemoryContextReset
  0.09%  postgres      [.] pgstat_end_function_usage
  0.08%  libc-2.12.so  [.] strcoll
  0.08%  postgres      [.] heap_hot_search_buffer
  0.07%  postgres      [.] lc_collate_is_c
  0.06%  [kernel]      [k] sys_semtimedop
  0.06%  postgres      [.] heap_page_prune_opt
  0.05%  postgres      [.] slot_getsomeattrs
  0.05%  postgres      [.] heap_fill_tuple
  0.04%  postgres      [.] hash_search
  0.03%  postgres      [.] GetMemoryChunkSpace
  0.03%  postgres      [.] heap_form_minimal_tuple
  0.03%  [kernel]      [k] update_queue
  0.02%  postgres      [.] ReadBufferExtended
  0.02%  postgres      [.] memcpy@plt

It seems that the box is using __strcoll a lot. The query performance is down, while previously the box was able to sustain with ~20 clients right now it is hardly able to keep up with 5.

I am wondering why the root cause might be here.

Let me know if anybody has seen this before.

Regards,
Istvan

--

the sun shines for all