I'm seeking two +1s to apply some work I did and tested in staging to the production proxies. The gist is that we currently proxy persistent tcp connections for the fedmsg firehose through our proxies to busgateway01 (which these days numbers around 700 connections but could scale up unexpectedly). We suspect that this has been causing our vpn issues since all fedmsg traffic is sent to the proxies for every connection (700 times the traffic normally on the bus). The following patch 'turns on' a scheme where each proxy will run its own fedmsg-gateway slave. Haproxy connects remote requests to these slaves, and the slaves in turn connect just once to the master fedmsg-gateway on busgateway01. Traffic should then only be sent once to each proxy (over the vpn in most cases). I tested it pretty thoroughly in stg (removed the daemon and its config and re-ran puppet a few times to make sure things were put in place in the correct order). Patch attached. -Ralph
From e88462f501ff79bf1f346f717b453d7a58d72517 Mon Sep 17 00:00:00 2001
From: Ralph Bean <rbean@xxxxxxxxxx>
Date: Thu, 4 Sep 2014 20:02:01 +0000
Subject: [PATCH] Apply fedmsg-gateway-slave work to prod proxies.
---
manifests/services/proxy.pp | 4 +---
modules/haproxy/files/haproxy.cfg | 7 +++----
2 files changed, 4 insertions(+), 7 deletions(-)
diff --git a/manifests/services/proxy.pp b/manifests/services/proxy.pp
index 4f2c8d0..d73b957 100644
--- a/manifests/services/proxy.pp
+++ b/manifests/services/proxy.pp
@@ -20,9 +20,7 @@ class proxy {
varnish::varnish{ 'proxy.vcl': }
}
- if $environment == "staging" {
- include haproxy::fedmsg-gateway-slave
- }
+ include haproxy::fedmsg-gateway-slave
$fpo_ips = [
diff --git a/modules/haproxy/files/haproxy.cfg b/modules/haproxy/files/haproxy.cfg
index 76f05d3..981d366 100644
--- a/modules/haproxy/files/haproxy.cfg
+++ b/modules/haproxy/files/haproxy.cfg
@@ -202,9 +202,8 @@ listen fedmsg-websockets 0.0.0.0:9938
# This, unlike the websockets entry just above, is listening directly to the
# outside world with no stunnel inbetween.
-# Simply redirect tcp connections to busgateway01 on the same port. There, an
-# instance of fedmsg-gateway will be publishing every message that comes across
-# the bus internally. This is for consumption by desktop applications.
+# Simply redirect tcp connections to a local fedmsg-gateway slave. It should be
+# forwarding messages from the master gateway on busgateway01.
listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
mode tcp
balance roundrobin
@@ -212,7 +211,7 @@ listen fedmsg-raw-zmq-outbound 0.0.0.0:9940
timeout queue 5000
timeout server 86400000
timeout connect 86400000
- server busgateway01 busgateway01:9940 weight 1 maxconn 16384
+ server localhost localhost:9942 weight 1 maxconn 16384
# While the above fedmsg-raw-zmq-outbound forwards incoming connections to an
# instance of the "fedmsg-gateway" daemon (which pushes internal messages out),
--
1.7.2.1
Attachment:
pgpvh0ERCIt3g.pgp
Description: PGP signature
_______________________________________________ infrastructure mailing list infrastructure@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/infrastructure
