Hi SELinux developers,
I'm seeing "kernel: SELinux: failure in sel_netport_sid_slow(), unable to determine network port label" under certain conditions.
OS -
Linux test33 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.2.1511 (Core)
Versions -
libselinux-2.2.2-6.el7.x86_64
libselinux-devel-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
libsemanage-2.1.10-18.el7.x86_64
libsemanage-python-2.1.10-18.el7.x86_64
libsepol-2.1.9-3.el7.x86_64
libsepol-devel-2.1.9-3.el7.x86_64
selinux-policy-3.13.1-60.el7_2.3.noarch
selinux-policy-targeted-3.13.1-60.el7_2.3.noarch
Scenario -
I run two Ansible roles, one deploys JupyterHub (a simple python+node app) which binds to some ports, and another immediately after (same Ansible playbook) that deploys Apache HTTPd (acting as reverse proxy for JupyterHub).
The JupyterHub role starts the JupyterHub app (a systemd unit) as its _last_ task, and the reverse proxy role adjusts 2 SE booleans as its _first_ task.
What happens is the 2 SE booleans get adjusted while JupyterHub is still starting and trying to bind to its ports, which causes the SELinux error shown above (and below in logs).
My current workaround is simply to wait at the end of my JupyterHub role until it has fully bound to its ports and listening (just 2-3 seconds), then continue with the reverse proxy role, but I wonder if there's a potential problem / race condition in SELinux?
Here's the log:
-- Last task of Ansible JupyterHub role - which is to start its service (ignore the 'Stopped', that's due to Ansible's state=restarted, but the app is not running at this time, and has never run on this host) --
Apr 12 15:39:53 test33 ansible-<stdin>[14326]: Invoked with name=jupyterhub pattern=None enabled=True state=restarted sleep=None arguments= runlevel=default
Apr 12 15:39:53 test33 systemd[1]: Reloading.
Apr 12 15:39:53 test33 systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Apr 12 15:39:53 test33 systemd[1]: Stopped JupyterHub Service.
Apr 12 15:39:53 test33 systemd[1]: Started JupyterHub Service.
Apr 12 15:39:53 test33 systemd[1]: Starting JupyterHub Service...
-- Reverse proxy role comes in, setting 'httpd_can_network_connect' and 'httpd_can_network_relay' bools, but JupyterHub startup isn't done yet --
Apr 12 15:39:53 test33 ansible-<stdin>[14367]: Invoked with warn=True executable=None chdir=None _raw_params=setsebool -P httpd_can_network_connect 1 && setsebool -P httpd_can_network_relay 1 removes=None creates=None _uses_shell=True
Apr 12 15:39:54 test33 dbus[537]: avc: received policyload notice (seqno=4)
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=4)
Apr 12 15:39:54 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:54 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:54 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:54 test33 kernel: SELinux: 8 users, 105 roles, 4965 types, 304 bools, 1 sens, 1024 cats
Apr 12 15:39:54 test33 kernel: SELinux: 83 classes, 105676 rules
-- JupyterHub continues with its startup, but cannot bind with an odd memory error, and kernel reports SELinux network port label issue, which I'm assuming is the cause of JupyterHub failing to bind --
Apr 12 15:39:54 test33 jupyterhub[14352]: [I 2016-04-12 15:39:54.340 JupyterHub app:572] Writing cookie_secret to /etc/jupyterhub/jupyterhub_cookie_secret
Apr 12 15:39:54 test33 jupyterhub[14352]: [W 2016-04-12 15:39:54.426 JupyterHub app:292]
Apr 12 15:39:54 test33 jupyterhub[14352]: Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
Apr 12 15:39:54 test33 jupyterhub[14352]: Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.
Apr 12 15:39:54 test33 jupyterhub[14352]: [W 2016-04-12 15:39:54.438 JupyterHub app:685] No admin users, admin interface will be unavailable.
Apr 12 15:39:54 test33 jupyterhub[14352]: [W 2016-04-12 15:39:54.438 JupyterHub app:686] Add any administrative users to `c.Authenticator.admin_users` in config.
Apr 12 15:39:54 test33 jupyterhub[14352]: [I 2016-04-12 15:39:54.438 JupyterHub app:712] Not using whitelist. Any authenticated user will be allowed.
Apr 12 15:39:54 test33 kernel: SELinux: failure in sel_netport_sid_slow(), unable to determine network port label
Apr 12 15:39:54 test33 jupyterhub[14352]: [E 2016-04-12 15:39:54.452 JupyterHub app:1110] Failed to bind hub to http://127.0.0.1:8081/hub/
Apr 12 15:39:54 test33 jupyterhub[14352]: [E 2016-04-12 15:39:54.452 JupyterHub app:1178]
Apr 12 15:39:54 test33 jupyterhub[14352]: Traceback (most recent call last):
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/jupyterhub/app.py", line 1176, in launch_instance_async
Apr 12 15:39:54 test33 jupyterhub[14352]: yield self.start()
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/jupyterhub/app.py", line 1108, in start
Apr 12 15:39:54 test33 jupyterhub[14352]: self.http_server.listen(self.hub_port, address=self.hub_ip)
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/tornado/tcpserver.py", line 126, in listen
Apr 12 15:39:54 test33 jupyterhub[14352]: sockets = bind_sockets(port, address=address)
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/tornado/netutil.py", line 196, in bind_sockets
Apr 12 15:39:54 test33 jupyterhub[14352]: sock.bind(sockaddr)
Apr 12 15:39:54 test33 jupyterhub[14352]: OSError: [Errno 12] Cannot allocate memory
Apr 12 15:39:54 test33 systemd[1]: jupyterhub.service: main process exited, code=exited, status=1/FAILURE
Apr 12 15:39:54 test33 systemd[1]: Unit jupyterhub.service entered failed state.
Apr 12 15:39:54 test33 systemd[1]: jupyterhub.service failed.
-- The 2 SE bools are set, not sure why there appears to be a reload every time and whether that's standard --
Apr 12 15:39:54 test33 dbus[537]: avc: received policyload notice (seqno=5)
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=5)
Apr 12 15:39:54 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:55 test33 setsebool[14369]: The httpd_can_network_connect policy boolean was changed to 1 by root
Apr 12 15:39:55 test33 dbus[537]: avc: received policyload notice (seqno=6)
Apr 12 15:39:55 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=6)
Apr 12 15:39:55 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:55 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:55 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:55 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:55 test33 kernel: SELinux: 8 users, 105 roles, 4965 types, 304 bools, 1 sens, 1024 cats
Apr 12 15:39:55 test33 kernel: SELinux: 83 classes, 105676 rules
Apr 12 15:39:56 test33 dbus[537]: avc: received policyload notice (seqno=7)
Apr 12 15:39:56 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=7)
Apr 12 15:39:56 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:56 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:56 test33 setsebool[14379]: The httpd_can_network_relay policy boolean was changed to 1 by root
As mentioned above, if I let JupyterHub fully bind to its ports before 'setsebool' is issued in the follow-up Ansible role, everything works absolutely fine, including restarting JupyterHub, and it works after rebooting etc, all healthy and normal and no SELinux errors.
Thanks,
Greg Kubok.
I'm seeing "kernel: SELinux: failure in sel_netport_sid_slow(), unable to determine network port label" under certain conditions.
OS -
Linux test33 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.2.1511 (Core)
Versions -
libselinux-2.2.2-6.el7.x86_64
libselinux-devel-2.2.2-6.el7.x86_64
libselinux-python-2.2.2-6.el7.x86_64
libselinux-utils-2.2.2-6.el7.x86_64
libsemanage-2.1.10-18.el7.x86_64
libsemanage-python-2.1.10-18.el7.x86_64
libsepol-2.1.9-3.el7.x86_64
libsepol-devel-2.1.9-3.el7.x86_64
selinux-policy-3.13.1-60.el7_2.3.noarch
selinux-policy-targeted-3.13.1-60.el7_2.3.noarch
SE Status -
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
Scenario -
I run two Ansible roles, one deploys JupyterHub (a simple python+node app) which binds to some ports, and another immediately after (same Ansible playbook) that deploys Apache HTTPd (acting as reverse proxy for JupyterHub).
The JupyterHub role starts the JupyterHub app (a systemd unit) as its _last_ task, and the reverse proxy role adjusts 2 SE booleans as its _first_ task.
What happens is the 2 SE booleans get adjusted while JupyterHub is still starting and trying to bind to its ports, which causes the SELinux error shown above (and below in logs).
My current workaround is simply to wait at the end of my JupyterHub role until it has fully bound to its ports and listening (just 2-3 seconds), then continue with the reverse proxy role, but I wonder if there's a potential problem / race condition in SELinux?
Here's the log:
-- Last task of Ansible JupyterHub role - which is to start its service (ignore the 'Stopped', that's due to Ansible's state=restarted, but the app is not running at this time, and has never run on this host) --
Apr 12 15:39:53 test33 ansible-<stdin>[14326]: Invoked with name=jupyterhub pattern=None enabled=True state=restarted sleep=None arguments= runlevel=default
Apr 12 15:39:53 test33 systemd[1]: Reloading.
Apr 12 15:39:53 test33 systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Apr 12 15:39:53 test33 systemd[1]: Stopped JupyterHub Service.
Apr 12 15:39:53 test33 systemd[1]: Started JupyterHub Service.
Apr 12 15:39:53 test33 systemd[1]: Starting JupyterHub Service...
-- Reverse proxy role comes in, setting 'httpd_can_network_connect' and 'httpd_can_network_relay' bools, but JupyterHub startup isn't done yet --
Apr 12 15:39:53 test33 ansible-<stdin>[14367]: Invoked with warn=True executable=None chdir=None _raw_params=setsebool -P httpd_can_network_connect 1 && setsebool -P httpd_can_network_relay 1 removes=None creates=None _uses_shell=True
Apr 12 15:39:54 test33 dbus[537]: avc: received policyload notice (seqno=4)
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=4)
Apr 12 15:39:54 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:54 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:54 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:54 test33 kernel: SELinux: 8 users, 105 roles, 4965 types, 304 bools, 1 sens, 1024 cats
Apr 12 15:39:54 test33 kernel: SELinux: 83 classes, 105676 rules
-- JupyterHub continues with its startup, but cannot bind with an odd memory error, and kernel reports SELinux network port label issue, which I'm assuming is the cause of JupyterHub failing to bind --
Apr 12 15:39:54 test33 jupyterhub[14352]: [I 2016-04-12 15:39:54.340 JupyterHub app:572] Writing cookie_secret to /etc/jupyterhub/jupyterhub_cookie_secret
Apr 12 15:39:54 test33 jupyterhub[14352]: [W 2016-04-12 15:39:54.426 JupyterHub app:292]
Apr 12 15:39:54 test33 jupyterhub[14352]: Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
Apr 12 15:39:54 test33 jupyterhub[14352]: Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.
Apr 12 15:39:54 test33 jupyterhub[14352]: [W 2016-04-12 15:39:54.438 JupyterHub app:685] No admin users, admin interface will be unavailable.
Apr 12 15:39:54 test33 jupyterhub[14352]: [W 2016-04-12 15:39:54.438 JupyterHub app:686] Add any administrative users to `c.Authenticator.admin_users` in config.
Apr 12 15:39:54 test33 jupyterhub[14352]: [I 2016-04-12 15:39:54.438 JupyterHub app:712] Not using whitelist. Any authenticated user will be allowed.
Apr 12 15:39:54 test33 kernel: SELinux: failure in sel_netport_sid_slow(), unable to determine network port label
Apr 12 15:39:54 test33 jupyterhub[14352]: [E 2016-04-12 15:39:54.452 JupyterHub app:1110] Failed to bind hub to http://127.0.0.1:8081/hub/
Apr 12 15:39:54 test33 jupyterhub[14352]: [E 2016-04-12 15:39:54.452 JupyterHub app:1178]
Apr 12 15:39:54 test33 jupyterhub[14352]: Traceback (most recent call last):
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/jupyterhub/app.py", line 1176, in launch_instance_async
Apr 12 15:39:54 test33 jupyterhub[14352]: yield self.start()
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/jupyterhub/app.py", line 1108, in start
Apr 12 15:39:54 test33 jupyterhub[14352]: self.http_server.listen(self.hub_port, address=self.hub_ip)
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/tornado/tcpserver.py", line 126, in listen
Apr 12 15:39:54 test33 jupyterhub[14352]: sockets = bind_sockets(port, address=address)
Apr 12 15:39:54 test33 jupyterhub[14352]: File "/usr/local/anaconda3/lib/python3.5/site-packages/tornado/netutil.py", line 196, in bind_sockets
Apr 12 15:39:54 test33 jupyterhub[14352]: sock.bind(sockaddr)
Apr 12 15:39:54 test33 jupyterhub[14352]: OSError: [Errno 12] Cannot allocate memory
Apr 12 15:39:54 test33 systemd[1]: jupyterhub.service: main process exited, code=exited, status=1/FAILURE
Apr 12 15:39:54 test33 systemd[1]: Unit jupyterhub.service entered failed state.
Apr 12 15:39:54 test33 systemd[1]: jupyterhub.service failed.
-- The 2 SE bools are set, not sure why there appears to be a reload every time and whether that's standard --
Apr 12 15:39:54 test33 dbus[537]: avc: received policyload notice (seqno=5)
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=5)
Apr 12 15:39:54 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:54 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:55 test33 setsebool[14369]: The httpd_can_network_connect policy boolean was changed to 1 by root
Apr 12 15:39:55 test33 dbus[537]: avc: received policyload notice (seqno=6)
Apr 12 15:39:55 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=6)
Apr 12 15:39:55 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:55 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:55 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:55 test33 kernel: SELinux: 2048 avtab hash slots, 105676 rules.
Apr 12 15:39:55 test33 kernel: SELinux: 8 users, 105 roles, 4965 types, 304 bools, 1 sens, 1024 cats
Apr 12 15:39:55 test33 kernel: SELinux: 83 classes, 105676 rules
Apr 12 15:39:56 test33 dbus[537]: avc: received policyload notice (seqno=7)
Apr 12 15:39:56 test33 dbus-daemon[537]: dbus[537]: avc: received policyload notice (seqno=7)
Apr 12 15:39:56 test33 dbus[537]: [system] Reloaded configuration
Apr 12 15:39:56 test33 dbus-daemon[537]: dbus[537]: [system] Reloaded configuration
Apr 12 15:39:56 test33 setsebool[14379]: The httpd_can_network_relay policy boolean was changed to 1 by root
As mentioned above, if I let JupyterHub fully bind to its ports before 'setsebool' is issued in the follow-up Ansible role, everything works absolutely fine, including restarting JupyterHub, and it works after rebooting etc, all healthy and normal and no SELinux errors.
Also, there are no policy violations of any kind when JupyterHub fails to bind, all looks clean except the kernel message in the log an ultimate failure to run:
# ausearch -m avc
<no matches>
Thanks,
Greg Kubok.
_______________________________________________ Selinux mailing list Selinux@xxxxxxxxxxxxx To unsubscribe, send email to Selinux-leave@xxxxxxxxxxxxx. To get help, send an email containing "help" to Selinux-request@xxxxxxxxxxxxx.
