This looks sensible. If the outage causes further issues we can do
'drastic' action then.
On 21 March 2017 at 16:52, Kevin Fenzi <kevin@xxxxxxxxx> wrote:
> Greetings.
>
> This thursday (2017-03-23) there's going to be some upgrades done on our
> netapp storage. These upgrades should not cause any outages if
> everything was working as expected, but there's an issue: A number of
> our machines are logged into iscsi luns, but not properly multipathing
> that connection, which means that those machines could see an outage to
> the iscsi storage since they don't have the redundant paths.
>
> However, we are not using iscsi as widely as we used to either, so a
> number of these machines are logged in, but don't otherwise use the
> storage at all.
>
> So, I'd like to get a freeze break to do the following:
>
> 1. Wait until our f26rc2 is composed (which might mean waiting until
> tomorrow).
>
> 2. Apply the following ansible patch:
>
> diff --git a/playbooks/groups/virthost.yml b/playbooks/groups/virthost.yml
> index e3d673f..e53d540 100644
> --- a/playbooks/groups/virthost.yml
> +++ b/playbooks/groups/virthost.yml
> @@ -22,7 +22,7 @@
> - hosts
> - fas_client
> - collectd/base
> - - { role: iscsi_client, when: datacenter == "phx2" }
> + - { role: iscsi_client, when:
> inventory_hostname.startswith(('bvirthost','buildvmhost')) }
> - sudo
> - { role: openvpn/client, when: datacenter != "phx2" }
> - virthost
>
> This will restrict the iscsi client setup to only bvirthost and
> buildvmhost machines. Those are the only ones that _should_ be using
> this storage (but see below):
>
> 3. Run ansible commands over the non bvirthost/buildvmhost machines to
> do a 'iscsi logout', disable iscsi and multipathd services and remove
> the /var/lib/iscsi/nodes/* files.
>
> 4. Re-provision docker-registry02.stg.phx2.fedoraproject.org and
> loopabull01.stg.phx2.fedoraproject.org that were provisioned on the
> iscsi storage and make them use local storage.
>
> 5. Sadly, that leaves bvirthost01/02/03 still not working right.
> However, they use 0 things from the iscsi storage, so we can fix them
> after freeze.
>
> That should hopefully do it. ;)
>
> +1s?
>
> kevin
>
>
> _______________________________________________
> infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
>
--
Stephen J Smoogen.
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
