Re: Announcing Dwarf: OpenStack API on top of libvirt/kvm

[Juerg Haefliger <juergh@xxxxxxxxx>]

Thanks for testing!


On Fri, Apr 11, 2014 at 11:21 AM, Sandro "red" Mathys <red@xxxxxxxxxxxxxxxxx> wrote:
>
> On Fri, Apr 11, 2014 at 2:53 PM, Sandro "red" Mathys
> <red@xxxxxxxxxxxxxxxxx> wrote:
> > On Fri, Apr 11, 2014 at 12:43 AM, Mats Wichmann <mats@xxxxxxxxxxx> wrote:
> >> On 04/10/14 02:09, Juerg Haefliger wrote:
> >>> In a nutshell, dwarf is OpenStack API on top of local libvirt/KVM.
> >>
> >> I suppose it's too late to suggest a name change?  dwarf already has
> >> long-time usage as the debugging format for various compilers, and
> >> there's a matching libdwarf with an api for examining the format.
> >> http://dwarfstd.org/
> >
> > FWIW: It's also not very ideal because one of the OpenStack components
> > (I think it was Trove) used to be called Red Dwarf...
> >
> > Anyway, thanks Juerg for packaging this and putting it into a copr
> > repo as requested. Will give it a try.
> >
> > -- Sandro
>
> Let me start by saying I did not see any SELinux issues, before I get
> to the bad (possibly TL;DR) news.
>
> So, on the only system I could try this out, a few things are changed
> from the defaults. I hope that didn't cause any of the issues I've
> found. I hope I didn't either. ;)
>
> If dwarf is started through systemd, it will log "sudo: sorry, you
> must have a tty to run sudo" whenever it is:
>
> 1) trying to chown the console-log - and therefore, 'nova console-log
> <server>' won't work:
> 2014-04-11 16:53:23,077 - INFO - dwarf.utils : execute(cmd=['chown',
> 991, '/var/lib/dwarf/instances/997adba9-ba7c-4f2a-9fff-94e4cfe5d37c/console.log'],
> check_exit_code=None, shell=False, run_as_root=True)
> 2014-04-11 16:53:23,106 - WARNING - dwarf.exception : Exception
> caught: Failed to run command: ['sudo', 'chown', '991',
> '/var/lib/dwarf/instances/997adba9-ba7c-4f2a-9fff-94e4cfe5d37c/console.log']
> exit_code: 1
> stdout:
> stderr: sudo: sorry, you must have a tty to run sudo
>  (500)
>
> 2) trying to set up the nat prerouting rule for the ec2 metadata service:
> 2014-04-11 16:27:22,717 - INFO - dwarf.utils :
> execute(cmd=['iptables', '-t', 'nat', '-D', 'PREROUTING', '-s', u'',
> '-d', '169.254.169.254/32', '-p', 'tcp', '-m', 'tcp', '--dport', 80,
> '-j', 'REDIRECT', '--to-port', 8080], check_exit_code=False,
> shell=False, run_as_root=True)
> 2014-04-11 16:27:22,729 - INFO - dwarf.utils : execute(stderr=sudo:
> sorry, you must have a tty to run sudo
> )
>
> ...now, working around the issue by starting dwarf by hand instead of
> through systemd should solve both based on the given sudo errors.

I admit I don't understand this. Starting it through systemd works just fine in my env (Fedora cloud image).

> But
> it seems the iptables still isn't put into place (but the log shows no
> issues). At least iptables -L -t nat shows no rules at all (yes, I
> cleared them beforehand). Now, I noticed that the above log entry
> shows "-s u" and wonder if "u" is ever replaced but since it's not
> logged anymore, I can't tell.

-s is the source IP, i.e., the IP that your instance should receive from dnsmasq. Did it get an IP at all?

You should see something like the following in the log:

2014-03-13 13:38:29,201 - INFO - dwarf.compute.servers : _get_server_ip(mac=52:54:00:f7:db:92) : None
2014-03-13 13:38:31,203 - INFO - dwarf.compute.servers : _get_server_ip(mac=52:54:00:f7:db:92) : None
2014-03-13 13:38:33,204 - INFO - dwarf.compute.servers : _get_server_ip(mac=52:54:00:f7:db:92) : None
2014-03-13 13:38:35,205 - INFO - dwarf.compute.servers : _get_server_ip(mac=52:54:00:f7:db:92) : None
2014-03-13 13:38:37,207 - INFO - dwarf.compute.servers : _get_server_ip(mac=52:54:00:f7:db:92) : 192.168.122.157

> Since that would result in an error I
> figure it would turn up in the log if it was an issue. Anyway, because
> of this cloud-init obviously can't reach the ec2 metadata. I tried to
> execute the iptables command myself but didn't figure out what I
> should replace u by. My guesses were either the instance's IP or
> virbr0's IP or the network address but all failed like:
> $ sudo iptables -t nat -D PREROUTING -s 192.168.111.1/24 -d
> 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-port 8080
> iptables: No chain/target/match by that name.
>
> (Yes, my libvirt is configured to use non-default IP ranges and yes, I
> updated dwarf.conf accordingly). That's where my poor network-fu ends.
>

> The work around successfully enables the console-log, though.

>
> Also, when trying to work around this issue by configuring the ec2
> metadata to listen on port 80 directly (and opening that port in the
> firewall - or just shut it down as I did) and waiting for cloud-init
> to fall back to the gateway IP instead of using 169.254.169.254, it
> will be able to connect but get a 404 not found. I'd expect this to
> work but either you really expect access through the special IP or you
> don't expect a request for /latest/meta-data/instance-id but only for
> /2009-04-04/meta-data-instance-id. I removed the workaround to try
> other things before I came up with this idea and therefore couldn't
> verify it right now.

I'm not a networking expert, obviously :-) The only way that I found for the ec2 metadata server to function in this environment was to attach it to the libvirt bridge's IP. The port doesn't really matter, as long as it's not occupied by something else.

/latest/<foo> should work.

...Juerg

> -- Sandro
> _______________________________________________
> cloud mailing list
> cloud@xxxxxxxxxxxxxxxxxxxxxxx
> https://admin.fedoraproject.org/mailman/listinfo/cloud
> Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

_______________________________________________
cloud mailing list
cloud@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/cloud
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct