Conntrackd Fail Over Causing loss of network traffic

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Kernel: 3.18.46
Connection tracking userspace daemon v1.4.2.

We use conntrackd for our Linux routers and have noticed new behavior when failing between the two devices. We track the resource using pacemaker and when we attempt to fail between the two devices, we get into a state where CPU usage for the core assigned to conntrackd will spike to 100%, traffic to the new device will be dropped and the process never seems to complete. In order to get back to a working state we'll need to fail back to the original device which was carrying the resource.

The only error we see in the logs is:  [warning] netlink event socket buffer size cannot be doubled further since it will exceed NetlinkBufferSizeMaxGrowth. We are likely to be losing events, this may lead to unsynchronized replica│·····
s. Please, consider increasing netlink socket buffer size via NetlinkBufferSize and NetlinkBufferSizeMaxGrowth clauses in conntrackd.conf 

I've increase the NetlinkBuffer size from 2 MB to 8 MB and the MaxGrowth from 8 MB to 16MB but I'm still seeing the same error when I attempt to fail over. I was hoping this list would be able to answer a few questions I had as to the behavior of conntrackd.

1. How does the loading of the connections from conntrackd to the netfliter hooks work in terms of the kernel? Is this operation locking (which would explain why we see new connections simply disappearing during this process)? Does netfilter dump all existing connections and then replace them with the new connections?

2. I set NetlinkOverrunResync off in the configuration as well as Filter From Kernelspace to improve performance and also hopefully alleviate the burden of exceeding the Netlink Buffer Size. Is my understanding correct that in situations where the buffer is overrun this prevents the forced resync against the kernel and the performance hit this presents?

3. What are the max sizes I can set for the buffer? Also with the buffer, what is the performance hit of having a buffer and then a max size buffer? Am I taking a performance hit by taking the first buffer and then attempting to copy it to a second buffer?

Thank for you for time and, as always, if I've missed some documentation online which explains all of these questions let me know. 

-Mat




[Index of Archives]     [Linux Netfilter Development]     [Linux Kernel Networking Development]     [Netem]     [Berkeley Packet Filter]     [Linux Kernel Development]     [Advanced Routing & Traffice Control]     [Bugtraq]

  Powered by Linux