Remote connect using virsh qemu+ssh hangs / PolicyKit issue

[Trey Dockendorf <treydock@xxxxxxxxx>]

I'm attempting to remote connect to my KVM instance using virsh, but all the commands hang.

When issuing the below command, nothing on the remote system happens, and no errors are displayed, (hostname changed)

$ virsh --debug 5 --log /var/lib/foreman/virsh.log -c qemu+ssh://foreman@xxxxxxxxxxx:16509/system?no_tty=1

This is the uncommented lines in /etc/libvirt/libvirtd.conf

----------

listen_tls = 0

listen_tcp = 1

listen_addr = "<omitted, set to management NIC>"

log_level = 1

log_filters="1:remote 1:event 1:qemu"

log_outputs="1:syslog:libvirtd 1:file:/var/log/libvirt/libvirtd.log"

This is the only debug output I get in /var/log/libvirt/libvirtd.log during the remote connection attempt

-----------

17:56:04.579: debug : virEventRunOnce:595 : Poll got 1 event

17:56:04.580: debug : virEventDispatchTimeouts:405 : Dispatch 3

17:56:04.580: debug : virEventDispatchHandles:450 : Dispatch 10

17:56:04.580: debug : virEventDispatchHandles:464 : i=0 w=1

17:56:04.580: debug : virEventDispatchHandles:464 : i=1 w=2

17:56:04.580: debug : virEventDispatchHandles:464 : i=2 w=3

17:56:04.580: debug : virEventDispatchHandles:464 : i=3 w=4

17:56:04.580: debug : virEventDispatchHandles:464 : i=4 w=5

17:56:04.580: debug : virEventDispatchHandles:464 : i=5 w=6

17:56:04.580: debug : virEventDispatchHandles:464 : i=6 w=7

17:56:04.580: debug : virEventDispatchHandles:464 : i=7 w=8

17:56:04.580: debug : virEventDispatchHandles:477 : Dispatch n=7 f=13 w=8 e=1 0x1629640

17:56:04.580: debug : virEventAddHandleImpl:113 : Add handle fd=20 events=1 cb=0x4196e0 opaque=0x1629640

17:56:04.580: debug : virEventInterruptLocked:664 : Skip interrupt, 1 -1447459072

17:56:04.580: debug : virEventDispatchHandles:464 : i=8 w=9

17:56:04.580: debug : virEventDispatchHandles:464 : i=9 w=10

17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3

17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11

17:56:04.580: debug : virEventCleanupTimeouts:495 : Cleanup 3

17:56:04.580: debug : virEventCleanupHandles:536 : Cleanupo 11

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=0 w=1, f=5 e=1

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=1 w=2, f=7 e=1

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=2 w=3, f=14 e=1

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=3 w=4, f=15 e=1

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=4 w=5, f=17 e=25

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=5 w=6, f=18 e=25

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=6 w=7, f=19 e=25

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=7 w=8, f=13 e=25

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=8 w=9, f=12 e=25

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=9 w=10, f=11 e=25

17:56:04.580: debug : virEventMakePollFDs:373 : Prepare n=10 w=15, f=20 e=1

17:56:04.580: debug : virEventCalculateTimeout:314 : Calculate expiry of 3 timers

17:56:04.580: debug : virEventCalculateTimeout:344 : Timeout at 0 due in -1 ms

17:56:04.580: debug : virEventRunOnce:593 : Poll on 11 handles 0x7f35a4001240 timeout -1

I've already opened up the firewall for port 16509, and allowed the user foreman (member of libvirt_admin) to manage libvirt via PolicyKit

Relevant line in iptables,

5    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:16509 

/etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla

-----------

[libvirt Remote Access]

Identity=unix-group:libvirt_admin 

Action="">

ResultAny=yes

ResultInactive=yes

ResultActive=yes

Originally I had created the file /etc/polkit-1/localauthority/50-local.d/51-libvirt-foreman-remote-access.pkla with contents below, and had the file 50-libvirt-remote-access.pkla only allowing a single user.

/etc/polkit-1/localauthority/50-local.d/51-libvirt-foreman-remote-access.pkla

----------

[libvirt Foreman Remote Access]

Identity=unix-user:foreman 

Action="">

ResultAny=yes

ResultInactive=yes

ResultActive=yes

However I wasn't able to connect to libvirt on the host itself, and the logs indicated it was a PolicyKit block, so my second problem/question...Is it possible to have multiple local PolicyKit *.pkla files or can only one exist?  From the documentation here, http://wiki.libvirt.org/page/SSHPolicyKitSetup, it seems like so long as the names are unique then multiple would be allowed.  Reason that's key is I'm using Puppet and will have multiple servers/applications needing access and being restricted to a single file to manage will be a problem.

Connecting locally with a specific pkla for "foreman"...

-----------

$ virsh -c qemu:///system

error: authentication failed

error: failed to connect to the hypervisor

/var/log/libvirt/libvirtd.log

---------

17:50:06.102: debug : virRunWithHook:914 : Command stderr: Not authorized.

17:50:06.103: error : remoteDispatchAuthPolkit:3810 : Policy kit denied action org.libvirt.unix.manage from pid 29640, uid 503, result: 256

Thanks

- Trey