Hi,
we are hitting the following kernel
issue, which we think might be caused by memory cgroups.:
[1.] One line summary of the problem:
Memory leaking when running kubernetes cronjobs
[2.] Full description of the problem/report:
We are using Kubernetes V1.8.15 with
docker 18.03.1-ce.
We schedule 50 Kubernetes cronjobs to
run every 5 minutes. Each cronjob will create a simple busybox container,
echo hello world, then terminate.
In the first set of data I have I let
this run for 1 hour, and in this time the Available memory had reduced
from 31256704 kB to 30461224 kB - so a loss of 776 MB.
There doesn't appear to be any processes
left behind, or any growth in any other processes to explain where the
memory has gone.
echo 3 > /proc/sys/vm/drop_caches
causes some of the memory to be returned, but the majority remains leaked,
and the only way to free it appears to be to reboot the system.
We are currently running Ubuntu 4.15.0-32.35-generic
4.15.18 and have previously observed similar issues on Ubuntu 16.04 with
Kernel 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 and
Debian 9.4 running 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02),
and I have just recreated on Linux version 4.19.0-041900rc3-generic
The leak was more severe on the Debian
system, and investigations there showed leaks in pcpu_get_vm_areas and
were related to memory cgroups. Running with Kernel 4.17 on debian showed
a leak at a similar rate to what we now observe on 4.15 & 4.19 Kernels
with Ubuntu. This leak causes us issues as we need to run cronjobs regularly
for backups and want the systems to remain up for months.
Kubernetes will create a new cgroup
each time the cronjob runs, but these are removed when the job completes
(which takes a few seconds). If I use systemd-cgtop I don't see any increase
in cgroups over time - but if I monitor /proc/cgroups over time I can see
num_cgroups for memory increases for about the first 18 hours, and then
stabilises at around 4300.
For the duration of the test I collected
slabinfo, meminfo, vmallocinfo & cgroups - which I can provide.
I stopped the cronjobs after 1 hour,
and I then left the system idle for 10 minutes. I then ran echo 3 >
/proc/sys/vm/drop_caches . This seemed to free ~240MB - but this still
leaves ~500MB lost. I then left the system idle for a further 20 minutes,
and MemoryAvailable didn't seem to be increasing significantly.
I have performed another run for a longer
period, and after 89 hours, the MemoryAvailable has reduced by ~5.5GB -
the rate of decrease seems less severe after the first 4-5 hours - but
clearly continues to decrease at a rate of ~1 GB per day for the duration
of the test.
After ~110 hours I ran `stress --vm
16 --vm-bytes 2147483648` to generate some memory pressure to see how much
would be reclaimed.
Before running the stress utility MemAvailable
was 24585576
After running the stress utility MemAvailable
was 27266788
I ran ps aux | awk '{sum+=$6} END {print
sum / 1024}'
which showed: 1220.44
Since the start of the test it looks
like we have leaked 3.8GB which is not reclaimable under memory pressure.
This was originally raised as this Ubuntu
bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1792349
, which requested a kernel bug be opened.
[3.] Keywords (i.e., modules, networking, kernel):
[4.] Kernel version (from /proc/version):Kernel
version: Linux version 4.19.0-041900rc3-generic (kernel@gloin) (gcc version
8.2.0 (Ubuntu 8.2.0-6ubuntu1)) #201809120832 SMP Wed Sep 12 12:35:08 UTC
2018
[5.] Output of Oops.. message (if applicable) with
symbolic information resolved (see Documentation/oops-tracing.txt):
n/a
[6.] A small shell script or example program which
triggers the problem (if possible):
Tricky to provide as Kubernetes is fairly complex to setup/configure
[7.] Environment
Description:
Ubuntu 18.04.1 LTS
Release:
18.04
[7.1.] Software (add the output of the ver_linux script
here):
Linux stage-dal09-carrier1-worker-37.alchemy.ibm.com
4.19.0-041900rc3-generic #201809120832 SMP Wed Sep 12 12:35:08 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
GNU Make
4.1
Binutils
2.30
Util-linux
2.31.1
Mount
2.31.1
Module-init-tools
24
E2fsprogs
1.44.1
Xfsprogs
4.9.0
Nfs-utils
1.3.3
Linux C Library
2.27
Dynamic linker (ldd)
2.27
Linux C++ Library
6.0.25
Procps
3.3.12
Net-tools
2.10
Kbd
2.0.4
Console-tools
2.0.4
Sh-utils
8.28
Udev
237
Modules Loaded
async_memcpy async_pq async_raid6_recov
async_tx async_xor autofs4 binfmt_misc bpfilter bridge btrfs cirrus crc32_pclmul
crct10dif_pclmul cryptd drm drm_kms_helper fb_sys_fops floppy ghash_clmulni_intel
hid hid_generic i2c_piix4 ib_cm ib_core ib_iser ide_core ide_pci_generic
input_leds intel_rapl intel_rapl_perf ip6_tables ip6t_REJECT ip6t_rt ip6table_filter
ip_set ip_set_hash_ip ip_set_hash_net ip_tables ip_tunnel ip_vs ipip ipt_MASQUERADE
ipt_REJECT iptable_filter iptable_mangle iptable_nat iptable_raw iscsi_tcp
iw_cm joydev libcrc32c libiscsi libiscsi_tcp linear llc mac_hid multipath
nf_conntrack nf_conntrack_broadcast nf_conntrack_ftp nf_conntrack_netbios_ns
nf_conntrack_netlink nf_defrag_ipv4 nf_defrag_ipv6 nf_log_common nf_log_ipv4
nf_log_ipv6 nf_nat nf_nat_ftp nf_nat_ipv4 nf_reject_ipv4 nf_reject_ipv6
nfnetlink overlay parport parport_pc pata_acpi piix ppdev psmouse raid0
raid1 raid10 raid456 raid6_pq rdma_cm sb_edac sch_fq_codel scsi_transport_iscsi
serio_raw stp sunrpc syscopyarea sysfillrect sysimgblt ttm tunnel4 usbhid
veth x_tables xen_privcmd xenfs xfrm_algo xfrm_user xor xt_LOG xt_addrtype
xt_comment xt_conntrack xt_hl xt_limit xt_mark xt_multiport xt_nat xt_recent
xt_set xt_statistic xt_tcpudp zstd_compress
[7.2.] Processor information (from /proc/cpuinfo):
(There are 16 processors - just including
first one for brevity)
processor :
0
vendor_id :
GenuineIntel
cpu family :
6
model
: 63
model name :
Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
stepping :
2
microcode :
0x3d
cpu MHz
: 2000.018
cache size :
35840 KB
physical id :
0
siblings :
16
core id
: 0
cpu cores :
16
apicid
: 0
initial apicid
: 0
fpu
: yes
fpu_exception :
yes
cpuid level :
13
wp
: yes
flags
: fpu vme de pse tsc msr pae mce cx8
apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ht
syscall pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid pni pclmulqdq
ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer
xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single
pti intel_ppin ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep
bmi2 erms invpcid xsaveopt flush_l1d
bugs
: cpu_meltdown spectre_v1 spectre_v2
spec_store_bypass l1tf
bogomips :
4000.08
clflush size :
64
cache_alignment
: 64
address sizes :
46 bits physical, 48 bits virtual
power management:
[7.3.] Module information (from /proc/modules):
binfmt_misc 20480 1 - Live 0xffffffffc0a7c000
veth 24576 0 - Live 0xffffffffc0a75000
xt_set 20480 4 - Live 0xffffffffc0a6f000
xt_multiport 16384 11 - Live 0xffffffffc0a65000
iptable_raw 16384 1 - Live 0xffffffffc0a60000
iptable_mangle 16384 1 - Live 0xffffffffc0a45000
ip_set_hash_ip 32768 1 - Live 0xffffffffc0a57000
ip_set_hash_net 32768 2 - Live 0xffffffffc0a4a000
ip_set 40960 3 xt_set,ip_set_hash_ip,ip_set_hash_net,
Live 0xffffffffc0a3a000
ipip 16384 0 - Live 0xffffffffc0a35000
tunnel4 16384 1 ipip, Live 0xffffffffc0a30000
ip_tunnel 24576 1 ipip, Live 0xffffffffc0a29000
xt_statistic 16384 5172 - Live 0xffffffffc0a24000
xt_nat 16384 8652 - Live 0xffffffffc0a1f000
xt_recent 20480 2 - Live 0xffffffffc0a15000
ipt_MASQUERADE 16384 3 - Live 0xffffffffc0a10000
xt_mark 16384 568 - Live 0xffffffffc096c000
bridge 159744 0 - Live 0xffffffffc09e8000
stp 16384 1 bridge, Live 0xffffffffc09e1000
llc 16384 2 bridge,stp, Live 0xffffffffc09d9000
xfrm_user 36864 1 - Live 0xffffffffc09cb000
xfrm_algo 16384 1 xfrm_user, Live 0xffffffffc09c6000
nf_conntrack_netlink 40960 0 - Live
0xffffffffc09bb000
nfnetlink 16384 3 ip_set,nf_conntrack_netlink,
Live 0xffffffffc0967000
xt_comment 16384 39264 - Live 0xffffffffc095c000
overlay 106496 8 - Live 0xffffffffc09a0000
iptable_nat 16384 1 - Live 0xffffffffc094e000
nf_nat_ipv4 16384 2 ipt_MASQUERADE,iptable_nat,
Live 0xffffffffc0957000
ip_vs 147456 0 - Live 0xffffffffc097b000
ppdev 20480 0 - Live 0xffffffffc0896000
intel_rapl 20480 0 - Live 0xffffffffc0975000
sb_edac 24576 0 - Live 0xffffffffc0947000
joydev 20480 0 - Live 0xffffffffc0961000
intel_rapl_perf 16384 0 - Live 0xffffffffc0891000
input_leds 16384 0 - Live 0xffffffffc088c000
serio_raw 16384 0 - Live 0xffffffffc0887000
parport_pc 36864 0 - Live 0xffffffffc08d4000
parport 49152 2 ppdev,parport_pc, Live
0xffffffffc08c2000
mac_hid 16384 0 - Live 0xffffffffc0882000
ip6t_REJECT 16384 1 - Live 0xffffffffc087a000
nf_reject_ipv6 16384 1 ip6t_REJECT,
Live 0xffffffffc0875000
nf_log_ipv6 16384 3 - Live 0xffffffffc0870000
xt_hl 16384 22 - Live 0xffffffffc086b000
ip6t_rt 16384 3 - Live 0xffffffffc0863000
ipt_REJECT 16384 23 - Live 0xffffffffc085b000
nf_reject_ipv4 16384 1 ipt_REJECT, Live
0xffffffffc0856000
nf_log_ipv4 16384 3 - Live 0xffffffffc084e000
nf_log_common 16384 2 nf_log_ipv6,nf_log_ipv4,
Live 0xffffffffc0846000
xt_LOG 16384 6 - Live 0xffffffffc083e000
xt_limit 16384 9 - Live 0xffffffffc082f000
xt_tcpudp 16384 20850 - Live 0xffffffffc082a000
xt_addrtype 16384 18 - Live 0xffffffffc0795000
sch_fq_codel 20480 17 - Live 0xffffffffc078f000
xt_conntrack 16384 155 - Live 0xffffffffc0822000
ip6table_filter 16384 1 - Live 0xffffffffc07c4000
ip6_tables 28672 53 ip6table_filter,
Live 0xffffffffc0816000
nf_conntrack_netbios_ns 16384 0 - Live
0xffffffffc080e000
nf_conntrack_broadcast 16384 1 nf_conntrack_netbios_ns,
Live 0xffffffffc073d000
nf_nat_ftp 16384 0 - Live 0xffffffffc06cc000
nf_nat 32768 3 xt_nat,nf_nat_ipv4,nf_nat_ftp,
Live 0xffffffffc079f000
nf_conntrack_ftp 20480 1 nf_nat_ftp,
Live 0xffffffffc0804000
nf_conntrack 143360 11 xt_nat,ipt_MASQUERADE,nf_conntrack_netlink,nf_nat_ipv4,ip_vs,xt_conntrack,nf_conntrack_netbios_ns,nf_conntrack_broadcast,nf_nat_ftp,nf_nat,nf_conntrack_ftp,
Live 0xffffffffc07d4000
nf_defrag_ipv6 20480 1 nf_conntrack,
Live 0xffffffffc07ca000
ib_iser 53248 0 - Live 0xffffffffc07b6000
nf_defrag_ipv4 16384 1 nf_conntrack,
Live 0xffffffffc0743000
rdma_cm 57344 1 ib_iser, Live 0xffffffffc072e000
iptable_filter 16384 1 - Live 0xffffffffc0729000
iw_cm 45056 1 rdma_cm, Live 0xffffffffc06b1000
ib_cm 53248 1 rdma_cm, Live 0xffffffffc07a8000
bpfilter 16384 0 - Live 0xffffffffc053a000
ib_core 237568 4 ib_iser,rdma_cm,iw_cm,ib_cm,
Live 0xffffffffc0754000
iscsi_tcp 20480 0 - Live 0xffffffffc074e000
libiscsi_tcp 20480 1 iscsi_tcp, Live
0xffffffffc0748000
sunrpc 356352 1 - Live 0xffffffffc06d1000
libiscsi 57344 3 ib_iser,iscsi_tcp,libiscsi_tcp,
Live 0xffffffffc06bd000
xenfs 16384 1 - Live 0xffffffffc0523000
xen_privcmd 20480 1 xenfs, Live 0xffffffffc0330000
scsi_transport_iscsi 98304 4 ib_iser,iscsi_tcp,libiscsi,
Live 0xffffffffc0698000
ip_tables 24576 14 iptable_raw,iptable_mangle,iptable_nat,iptable_filter,
Live 0xffffffffc045f000
x_tables 36864 23 xt_set,xt_multiport,iptable_raw,iptable_mangle,xt_statistic,xt_nat,xt_recent,ipt_MASQUERADE,xt_mark,xt_comment,ip6t_REJECT,xt_hl,ip6t_rt,ipt_REJECT,xt_LOG,xt_limit,xt_tcpudp,xt_addrtype,xt_conntrack,ip6table_filter,ip6_tables,iptable_filter,ip_tables,
Live 0xffffffffc068e000
autofs4 40960 2 - Live 0xffffffffc0683000
btrfs 1159168 0 - Live 0xffffffffc0567000
zstd_compress 159744 1 btrfs, Live 0xffffffffc053f000
raid10 53248 0 - Live 0xffffffffc052c000
raid456 151552 0 - Live 0xffffffffc04fd000
async_raid6_recov 20480 1 raid456, Live
0xffffffffc04f7000
async_memcpy 16384 2 raid456,async_raid6_recov,
Live 0xffffffffc04f2000
async_pq 16384 2 raid456,async_raid6_recov,
Live 0xffffffffc04ed000
async_xor 16384 3 raid456,async_raid6_recov,async_pq,
Live 0xffffffffc04b5000
async_tx 16384 5 raid456,async_raid6_recov,async_memcpy,async_pq,async_xor,
Live 0xffffffffc04b0000
xor 24576 2 btrfs,async_xor, Live 0xffffffffc0483000
raid6_pq 114688 4 btrfs,raid456,async_raid6_recov,async_pq,
Live 0xffffffffc0493000
libcrc32c 16384 5 ip_vs,nf_nat,nf_conntrack,btrfs,raid456,
Live 0xffffffffc047e000
raid1 40960 0 - Live 0xffffffffc046b000
raid0 20480 0 - Live 0xffffffffc0422000
multipath 16384 0 - Live 0xffffffffc02fe000
linear 16384 0 - Live 0xffffffffc0312000
hid_generic 16384 0 - Live 0xffffffffc0304000
usbhid 49152 0 - Live 0xffffffffc02f1000
hid 126976 2 hid_generic,usbhid, Live
0xffffffffc042e000
cirrus 24576 1 - Live 0xffffffffc02e4000
ttm 106496 1 cirrus, Live 0xffffffffc03eb000
drm_kms_helper 167936 1 cirrus, Live
0xffffffffc04c3000
syscopyarea 16384 1 drm_kms_helper,
Live 0xffffffffc02ec000
crct10dif_pclmul 16384 0 - Live 0xffffffffc04bb000
sysfillrect 16384 1 drm_kms_helper,
Live 0xffffffffc048e000
crc32_pclmul 16384 0 - Live 0xffffffffc0429000
ide_pci_generic 16384 0 - Live 0xffffffffc045a000
sysimgblt 16384 1 drm_kms_helper, Live
0xffffffffc0317000
ghash_clmulni_intel 16384 0 - Live 0xffffffffc0479000
fb_sys_fops 16384 1 drm_kms_helper,
Live 0xffffffffc0466000
piix 16384 0 - Live 0xffffffffc0452000
cryptd 24576 1 ghash_clmulni_intel,
Live 0xffffffffc0347000
ide_core 106496 2 ide_pci_generic,piix,
Live 0xffffffffc0407000
drm 466944 4 cirrus,ttm,drm_kms_helper,
Live 0xffffffffc0352000
psmouse 151552 0 - Live 0xffffffffc03c5000
i2c_piix4 24576 0 - Live 0xffffffffc033c000
pata_acpi 16384 0 - Live 0xffffffffc0337000
floppy 77824 0 - Live 0xffffffffc031c000
[7.4.] Loaded driver and hardware information (/proc/ioports,
/proc/iomem)
cat /proc/ioports
0000-0cf7 : PCI Bus 0000:00
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0061-0061 : PNP0800:00
0064-0064 : keyboard
0070-0071 : rtc0
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : 0000:00:01.1
0170-0177 : ata_piix
01f0-01f7 : 0000:00:01.1
01f0-01f7 : ata_piix
0376-0376 : 0000:00:01.1
0376-0376 : ata_piix
0378-037a : parport0
03c0-03df : vga+
03f2-03f2 : floppy
03f4-03f5 : floppy
03f6-03f6 : 0000:00:01.1
03f6-03f6 : ata_piix
03f7-03f7 : floppy
03f8-03ff : serial
04d0-04d1 : pnp 00:01
08a0-08a3 : pnp 00:01
0cc0-0ccf : pnp 00:01
0cf8-0cff : PCI conf1
0d00-ffff : PCI Bus 0000:00
10c0-1141 : pnp 00:08
afe0-afe3 : ACPI GPE0_BLK
b000-b03f : 0000:00:01.3
b000-b003 : ACPI PM1a_EVT_BLK
b004-b005 : ACPI PM1a_CNT_BLK
b008-b00b : ACPI PM_TMR
b044-b047 : pnp 00:08
c000-c0ff : 0000:00:03.0
c000-c0ff : xen-platform-pci
c300-c31f : 0000:00:01.2
c300-c31f : uhci_hcd
c320-c32f : 0000:00:01.1
c320-c32f : ata_piix
cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009dfff : System RAM
0009e000-0009ffff : Reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000c8bff : Video ROM
000c9000-000c99ff : Adapter ROM
000e0000-000fffff : Reserved
000f0000-000fffff : System ROM
00100000-efffffff : System RAM
f0000000-fbffffff : PCI Bus 0000:00
f0000000-f1ffffff : 0000:00:02.0
f0000000-f1ffffff : cirrusdrmfb_vram
f2000000-f2ffffff : 0000:00:03.0
f2000000-f2ffffff : xen-platform-pci
f3000000-f33fffff : 0000:00:06.0
f3400000-f3400fff : 0000:00:02.0
f3400000-f3400fff : cirrusdrmfb_mmio
fc000000-ffffffff : Reserved
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fee00000-fee00fff : Local APIC
100000000-80fbfffff : System RAM
30b400000-30c2031d0 : Kernel
code
30c2031d1-30cc9fa3f : Kernel
data
30cf22000-30d16dfff : Kernel
bss
80fc00000-80fffffff : RAM buffer
[7.5.] PCI information ('lspci -vvv' as root)
sudo lspci -vvv
00:00.0 Host bridge: Intel Corporation
440FX - 82441FX PMC [Natoma] (rev 02)
Subsystem:
Red Hat, Inc Qemu virtual machine
Physical
Slot: 0
Control:
I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-
>SERR- <PERR- INTx-
Latency:
0
00:01.0 ISA bridge: Intel Corporation
82371SB PIIX3 ISA [Natoma/Triton II]
Subsystem:
Red Hat, Inc Qemu virtual machine
Physical
Slot: 1
Control:
I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency:
0
00:01.1 IDE interface: Intel Corporation
82371SB PIIX3 IDE [Natoma/Triton II] (prog-if 80 [Master])
Subsystem:
XenSource, Inc. 82371SB PIIX3 IDE [Natoma/Triton II]
Physical
Slot: 1
Control:
I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency:
64
Region
0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
Region
1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
Region
2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
Region
3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
Region
4: I/O ports at c320 [size=16]
Kernel
driver in use: ata_piix
Kernel
modules: pata_acpi, piix, ide_pci_generic
00:01.2 USB controller: Intel Corporation
82371SB PIIX3 USB [Natoma/Triton II] (rev 01) (prog-if 00 [UHCI])
Subsystem:
XenSource, Inc. 82371SB PIIX3 USB [Natoma/Triton II]
Physical
Slot: 1
Control:
I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-
>SERR- <PERR- INTx-
Latency:
64
Interrupt:
pin D routed to IRQ 23
Region
4: I/O ports at c300 [size=32]
Kernel
driver in use: uhci_hcd
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB
PIIX4 ACPI (rev 01)
Subsystem:
Red Hat, Inc Qemu virtual machine
Physical
Slot: 1
Control:
I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-
>SERR- <PERR- INTx-
Latency:
0
Interrupt:
pin A routed to IRQ 9
Kernel
modules: i2c_piix4
00:02.0 VGA compatible controller: Cirrus
Logic GD 5446 (prog-if 00 [VGA controller])
Subsystem:
XenSource, Inc. GD 5446
Physical
Slot: 2
Control:
I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-
>SERR- <PERR- INTx-
Latency:
0
Interrupt:
pin A routed to IRQ 24
Region
0: Memory at f0000000 (32-bit, prefetchable) [size=32M]
Region
1: Memory at f3400000 (32-bit, non-prefetchable) [size=4K]
[virtual]
Expansion ROM at 000c0000 [disabled] [size=128K]
Kernel
driver in use: cirrus
Kernel
modules: cirrusfb, cirrus
00:03.0 SCSI storage controller: XenSource,
Inc. Xen Platform Device (rev 01)
Subsystem:
XenSource, Inc. Xen Platform Device
Physical
Slot: 3
Control:
I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-
>SERR- <PERR- INTx-
Latency:
0
Interrupt:
pin C routed to IRQ 30
Region
0: I/O ports at c000 [size=256]
Region
1: Memory at f2000000 (32-bit, prefetchable) [size=16M]
Kernel
driver in use: xen-platform-pci
00:06.0 System peripheral: XenSource,
Inc. Citrix XenServer PCI Device for Windows Update (rev 01)
Subsystem:
XenSource, Inc. Citrix XenServer PCI Device for Windows Update
Physical
Slot: 6
Control:
I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR-
FastB2B- DisINTx-
Status:
Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort-
>SERR- <PERR- INTx-
Latency:
0
Interrupt:
pin C routed to IRQ 5
Region
1: Memory at f3000000 (32-bit, prefetchable) [size=4M]
[7.6.] SCSI information (from /proc/scsi/scsi)
Attached devices:
[7.7.] Other information that might be relevant to
the problem
(please look in /proc and
include all information that you
think to be relevant):
cat slabinfo
slabinfo - version: 2.1
# name
<active_objs> <num_objs> <objsize> <objperslab>
<pagesperslab> : tunables <limit> <batchcount> <sharedfactor>
: slabdata <active_slabs> <num_slabs> <sharedavail>
ovl_inode
9408 9408 680 48 8 : tunables
0 0 0 : slabdata 196
196 0
ext4_groupinfo_1k 60
60 136 60 2 : tunables 0
0 0 : slabdata 1
1 0
nf_conntrack_expect 0
0 216 37 2 : tunables
0 0 0 : slabdata 0
0 0
nf_conntrack 6783
6783 320 51 4 : tunables
0 0 0 : slabdata 133
133 0
rpc_inode_cache 102
102 640 51 8 : tunables
0 0 0 : slabdata 2
2 0
ext4_groupinfo_4k 3248
3248 144 56 2 : tunables 0
0 0 : slabdata 58
58 0
btrfs_delayed_ref_head
0 0 152 53 2
: tunables 0 0 0 : slabdata
0 0 0
btrfs_delayed_node 0
0 296 55 4 : tunables
0 0 0 : slabdata 0
0 0
btrfs_ordered_extent 0
0 416 39 4 : tunables
0 0 0 : slabdata 0
0 0
btrfs_extent_map
0 0 144 56 2 : tunables
0 0 0 : slabdata 0
0 0
btrfs_extent_buffer 0
0 280 58 4 : tunables
0 0 0 : slabdata 0
0 0
btrfs_path
0 0 112 36
1 : tunables 0 0 0 : slabdata
0 0 0
btrfs_inode
0 0 1136 28 8
: tunables 0 0 0 : slabdata
0 0 0
PINGv6
0 0 1152 28
8 : tunables 0 0 0 : slabdata
0 0 0
RAWv6
2184 2408 1152 28 8 : tunables
0 0 0 : slabdata 86
86 0
UDPv6
4037 4175 1280 25 8 : tunables
0 0 0 : slabdata 167
167 0
tw_sock_TCPv6 2176
2176 240 68 4 : tunables
0 0 0 : slabdata 32
32 0
request_sock_TCPv6 0
0 304 53 4 : tunables
0 0 0 : slabdata 0
0 0
TCPv6
2716 2716 2304 14 8 : tunables
0 0 0 : slabdata 194
194 0
kcopyd_job
0 0 3312 9
8 : tunables 0 0 0 : slabdata
0 0 0
dm_uevent
0 0 2632 12
8 : tunables 0 0 0 : slabdata
0 0 0
scsi_sense_cache
0 0 128 64 2 : tunables
0 0 0 : slabdata 0
0 0
cfq_io_cq
0 0 120 68
2 : tunables 0 0 0
: slabdata 0 0 0
mqueue_inode_cache 544
544 960 34 8 : tunables
0 0 0 : slabdata 16
16 0
fuse_request
640 640 400 40 4 : tunables
0 0 0 : slabdata 16
16 0
fuse_inode
7566 7566 832 39 8 : tunables
0 0 0 : slabdata 194
194 0
ecryptfs_key_record_cache
0 0 576 56 8
: tunables 0 0 0 : slabdata
0 0 0
ecryptfs_headers
8 8 4096 8 8 : tunables
0 0 0 : slabdata 1
1 0
ecryptfs_inode_cache 0
0 960 34 8 : tunables
0 0 0 : slabdata 0
0 0
ecryptfs_dentry_info_cache 128
128 32 128 1 : tunables
0 0 0 : slabdata 1
1 0
ecryptfs_file_cache 0
0 16 256 1 : tunables
0 0 0 : slabdata 0
0 0
ecryptfs_auth_tok_list_item
0 0 832 39 8
: tunables 0 0 0 : slabdata
0 0 0
fat_inode_cache
0 0 736 44 8
: tunables 0 0 0 : slabdata
0 0 0
fat_cache
0 0 40 102
1 : tunables 0 0 0
: slabdata 0 0 0
squashfs_inode_cache 92
92 704 46 8 : tunables
0 0 0 : slabdata 2
2 0
jbd2_journal_head 12376 12376
120 68 2 : tunables 0
0 0 : slabdata 182 182
0
jbd2_revoke_table_s 768
768 16 256 1 : tunables
0 0 0 : slabdata 3
3 0
ext4_inode_cache 73909 82020
1080 30 8 : tunables 0 0
0 : slabdata 2734 2734 0
ext4_allocation_context 1024
1024 128 64 2 : tunables
0 0 0 : slabdata 16
16 0
ext4_io_end
1984 1984 64 64 1 : tunables
0 0 0 : slabdata 31
31 0
ext4_extent_status 82192 83130
40 102 1 : tunables 0
0 0 : slabdata 815 815
0
mbcache
5548 5548 56 73 1
: tunables 0 0 0 : slabdata
76 76 0
fscrypt_info 7680
7680 32 128 1 : tunables
0 0 0 : slabdata 60
60 0
fscrypt_ctx
1360 1360 48 85 1 : tunables
0 0 0 : slabdata 16
16 0
userfaultfd_ctx_cache
0 0 192 42 2
: tunables 0 0 0 : slabdata
0 0 0
dnotify_struct
0 0 32 128 1
: tunables 0 0 0 : slabdata
0 0 0
posix_timers_cache 544
544 240 68 4 : tunables
0 0 0 : slabdata 8
8 0
UNIX
8544 8544 1024 32 8
: tunables 0 0 0 : slabdata
267 267 0
ip4-frags
624 624 208 39 2
: tunables 0 0 0 : slabdata
16 16 0
secpath_cache 1024
1024 128 64 2 : tunables
0 0 0 : slabdata 16
16 0
xfrm_dst_cache
0 0 320 51 4
: tunables 0 0 0 : slabdata
0 0 0
xfrm_state
0 0 768 42
8 : tunables 0 0 0 : slabdata
0 0 0
PING
0 0 960
34 8 : tunables 0 0 0
: slabdata 0 0 0
RAW
6086 6392 960 34
8 : tunables 0 0 0 : slabdata
188 188 0
tw_sock_TCP
2176 2176 240 68 4 : tunables
0 0 0 : slabdata 32
32 0
request_sock_TCP 901
901 304 53 4 : tunables 0
0 0 : slabdata 17
17 0
TCP
1605 1605 2176 15 8
: tunables 0 0 0 : slabdata
107 107 0
hugetlbfs_inode_cache 104
104 624 52 8 : tunables
0 0 0 : slabdata 2
2 0
dquot
1024 1024 256 64 4
: tunables 0 0 0 : slabdata
16 16 0
eventpoll_pwq 25984
25984 72 56 1 : tunables
0 0 0 : slabdata 464
464 0
inotify_inode_mark 3978
3978 80 51 1 : tunables 0
0 0 : slabdata 78
78 0
dax_cache
42 42 768 42 8
: tunables 0 0 0 : slabdata
1 1 0
request_queue
39 39 2480 13 8 : tunables
0 0 0 : slabdata 3
3 0
blkdev_requests
52 52 312 52 4 : tunables
0 0 0 : slabdata 1
1 0
biovec-max
300 304 8192 4 8 :
tunables 0 0 0 : slabdata
76 76 0
biovec-128
32 32 2048 16 8 :
tunables 0 0 0 : slabdata
2 2 0
biovec-64
512 512 1024 32 8
: tunables 0 0 0 : slabdata
16 16 0
dmaengine-unmap-256 15
15 2112 15 8 : tunables
0 0 0 : slabdata 1
1 0
dmaengine-unmap-128 30
30 1088 30 8 : tunables
0 0 0 : slabdata 1
1 0
dmaengine-unmap-16 13860 13860
192 42 2 : tunables 0
0 0 : slabdata 330 330
0
dmaengine-unmap-2 3072
3072 64 64 1 : tunables 0
0 0 : slabdata 48
48 0
sock_inode_cache 26914 27232
704 46 8 : tunables 0
0 0 : slabdata 592 592
0
skbuff_fclone_cache 1216
1216 512 64 8 : tunables 0
0 0 : slabdata 19
19 0
skbuff_head_cache 33488 34368
256 64 4 : tunables 0
0 0 : slabdata 537 537
0
file_lock_cache 640
640 200 40 2 : tunables
0 0 0 : slabdata 16
16 0
fsnotify_mark_connector 2720
2720 24 170 1 : tunables
0 0 0 : slabdata 16
16 0
net_namespace 180
185 6336 5 8 : tunables
0 0 0 : slabdata 37
37 0
shmem_inode_cache 17822 18400
704 46 8 : tunables 0
0 0 : slabdata 400 400
0
taskstats
784 784 328 49 4
: tunables 0 0 0 : slabdata
16 16 0
proc_dir_entry 8603
8694 192 42 2 : tunables
0 0 0 : slabdata 207
207 0
pde_opener
22848 22848 40 102 1 : tunables
0 0 0 : slabdata 224
224 0
proc_inode_cache 49963 51888
672 48 8 : tunables 0
0 0 : slabdata 1081 1081
0
sigqueue
816 816 160 51 2
: tunables 0 0 0 : slabdata
16 16 0
bdev_cache
312 312 832 39 8 :
tunables 0 0 0 : slabdata
8 8 0
kernfs_node_cache 7775161 7775400
136 60 2 : tunables 0 0
0 : slabdata 129590 129590 0
mnt_cache
5215 5250 384 42 4 : tunables
0 0 0 : slabdata 125
125 0
filp
167618 171136 256 64 4 :
tunables 0 0 0 : slabdata
2674 2674 0
inode_cache 5517103
5523174 600 54 8 : tunables 0
0 0 : slabdata 102281 102281 0
dentry
14733403 14827218 192 42 2 : tunables
0 0 0 : slabdata 353029 353029
0
names_cache
296 296 4096 8 8 :
tunables 0 0 0 : slabdata
37 37 0
iint_cache
0 0 120 68
2 : tunables 0 0 0 : slabdata
0 0 0
buffer_head 2184093
2604927 104 39 1 : tunables 0
0 0 : slabdata 66793 66793
0
uts_namespace 592
592 440 37 4 : tunables
0 0 0 : slabdata 16
16 0
nsproxy
1752 1752 56 73 1
: tunables 0 0 0 : slabdata
24 24 0
vm_area_struct 108257 109080
200 40 2 : tunables 0
0 0 : slabdata 2727 2727
0
mm_struct
9330 9330 1088 30 8 : tunables
0 0 0 : slabdata 311
311 0
files_cache 11040
11040 704 46 8 : tunables
0 0 0 : slabdata 240
240 0
signal_cache 12270
12270 1088 30 8 : tunables 0
0 0 : slabdata 409 409
0
sighand_cache 4780
4800 2112 15 8 : tunables 0
0 0 : slabdata 320 320
0
task_struct
3625 3815 5888 5 8 : tunables
0 0 0 : slabdata 763
763 0
cred_jar
56532 56532 192 42 2 : tunables
0 0 0 : slabdata 1346
1346 0
anon_vma_chain 130241 132096
64 64 1 : tunables 0
0 0 : slabdata 2064 2064
0
anon_vma
83964 84364 88 46 1 : tunables
0 0 0 : slabdata 1834
1834 0
pid
41603 41728 128 64
2 : tunables 0 0 0 : slabdata
652 652 0
Acpi-Operand 4256
4256 72 56 1 : tunables
0 0 0 : slabdata 76
76 0
Acpi-ParseExt 624
624 104 39 1 : tunables
0 0 0 : slabdata 16
16 0
Acpi-State
459 459 80 51 1 :
tunables 0 0 0 : slabdata
9 9 0
Acpi-Namespace 4284
4284 40 102 1 : tunables
0 0 0 : slabdata 42
42 0
numa_policy
186 186 264 62 4 :
tunables 0 0 0 : slabdata
3 3 0
trace_event_file 2116
2116 88 46 1 : tunables 0
0 0 : slabdata 46
46 0
ftrace_event_field 24294 25670
48 85 1 : tunables 0
0 0 : slabdata 302 302
0
pool_workqueue 4736
4736 256 64 4 : tunables
0 0 0 : slabdata 74
74 0
radix_tree_node 2679706 2689960
584 56 8 : tunables 0
0 0 : slabdata 48035 48035
0
task_group
816 816 640 51 8 :
tunables 0 0 0 : slabdata
16 16 0
dma-kmalloc-8192
0 0 8192 4 8 : tunables
0 0 0 : slabdata 0
0 0
dma-kmalloc-4096
0 0 4096 8 8 : tunables
0 0 0 : slabdata 0
0 0
dma-kmalloc-2048
0 0 2048 16 8 : tunables
0 0 0 : slabdata 0
0 0
dma-kmalloc-1024
0 0 1024 32 8 : tunables
0 0 0 : slabdata 0
0 0
dma-kmalloc-512
0 0 512 64 8
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-256
0 0 256 64 4
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-128
0 0 128 64 2
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-64
0 0 64 64 1
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-32
0 0 32 128 1
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-16
0 0 16 256 1
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-8
0 0 8 512 1
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-192
0 0 192 42 2
: tunables 0 0 0 : slabdata
0 0 0
dma-kmalloc-96
0 0 96 42 1
: tunables 0 0 0 : slabdata
0 0 0
kmalloc-8192
827 844 8192 4 8 : tunables
0 0 0 : slabdata 211
211 0
kmalloc-4096 7301
8208 4096 8 8 : tunables
0 0 0 : slabdata 1026 1026
0
kmalloc-2048 9739
13440 2048 16 8 : tunables 0
0 0 : slabdata 840 840
0
kmalloc-1024 20177
22272 1024 32 8 : tunables 0
0 0 : slabdata 696 696
0
kmalloc-512 62892
94464 512 64 8 : tunables
0 0 0 : slabdata 1476 1476
0
kmalloc-256 22049
25024 256 64 4 : tunables
0 0 0 : slabdata 391
391 0
kmalloc-192 1296144
1296918 192 42 2 : tunables 0
0 0 : slabdata 30879 30879
0
kmalloc-128 7230544
7231680 128 64 2 : tunables 0
0 0 : slabdata 112995 112995 0
kmalloc-96 8202597
8204952 96 42 1 : tunables 0
0 0 : slabdata 195356 195356 0
kmalloc-64 2169882
2190016 64 64 1 : tunables 0
0 0 : slabdata 34219 34219
0
kmalloc-32 147668
303872 32 128 1 : tunables 0
0 0 : slabdata 2374 2374
0
kmalloc-16 156395
170496 16 256 1 : tunables 0
0 0 : slabdata 666 666
0
kmalloc-8
131584 131584 8 512 1 : tunables
0 0 0 : slabdata 257
257 0
kmem_cache_node 74067 74880
64 64 1 : tunables 0
0 0 : slabdata 1170 1170
0
kmem_cache
73491 74004 384 42 4 : tunables
0 0 0 : slabdata 1762
1762 0
cat meminfo
MemTotal: 32910232
kB
MemFree:
6742744 kB
MemAvailable: 25470032 kB
Buffers:
947924 kB
Cached:
8611528 kB
SwapCached:
0 kB
Active:
5668624 kB
Inactive: 5374688
kB
Active(anon): 1488612 kB
Inactive(anon): 620
kB
Active(file): 4180012 kB
Inactive(file): 5374068 kB
Unevictable: 5408
kB
Mlocked:
5408 kB
SwapTotal:
0 kB
SwapFree:
0 kB
Dirty:
520 kB
Writeback:
0 kB
AnonPages: 1478048
kB
Mapped:
305152 kB
Shmem:
1164 kB
Slab:
11596304 kB
SReclaimable: 8061752 kB
SUnreclaim: 3534552
kB
KernelStack: 14800
kB
PageTables: 13676
kB
NFS_Unstable:
0 kB
Bounce:
0 kB
WritebackTmp:
0 kB
CommitLimit: 16455116 kB
Committed_AS: 3378396 kB
VmallocTotal: 34359738367 kB
VmallocUsed:
0 kB
VmallocChunk:
0 kB
Percpu:
3316224 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10240 kB
ShmemHugePages:
0 kB
ShmemPmdMapped:
0 kB
CmaTotal:
0 kB
CmaFree:
0 kB
HugePages_Total:
0
HugePages_Free:
0
HugePages_Rsvd:
0
HugePages_Surp:
0
Hugepagesize: 2048
kB
Hugetlb:
0 kB
DirectMap4k: 387072
kB
DirectMap2M: 27920384 kB
DirectMap1G: 6291456 kB
Dan McGinnes
IBM Cloud - Containers performance
Int Tel: 247359 Ext Tel: 01962 817359
Notes: Daniel McGinnes/UK/IBM
Email: MCGINNES@xxxxxxxxxx
IBM (UK) Ltd, Hursley Park,Winchester,Hampshire, SO21 2JN
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
