Simon Waters wrote:
One of our firewalls appears to be unstable.
The problems started with adding ip_conntrack and ip_conntrack_ftp, as we need
to support active mode FTP from inside to the outside.
Prior to this we were using it as a basic port filter (poor iptables), and so
many of the rules exist to allow return packets without connection tracking.
As such it makes sense for us to "clean-up" the ruleset, but this hasn't
happened yet.
Due to some interesting (and maybe related?) hardware issues, we've run the
same firewall configuration on three different servers (all x86, Redhat and
Debian, IDE and SCSI), including one with the latest (for Debian) 2.6.8
kernel, and are fairly confident we see the same software issue on all three
boxes.
Symptom is that by the time we get to it, the box is totally unresponsive to
local console, is not forwarding packets. In a word "hung".
Memory isn't obviously leaking.
I do not recall what 2.6 kernel it was in, but there was a memory leak that plagued a system that did (extensive?) firewalling / routing in the mid 2.6 kernels. I ran in to this on a system and ended up croning a reboot daily. I have not had time to go back and upgrade the system to a kernel that does not have the memory leak. I am not sure but I think I have 2.6.8 on the box.
The number of lines in "ip_conntrack" does appear to grow with time, but is
still way below (at around 3000) the maximum allowed of 32,000+, and isn't
growing monotonically. I think there may be clues here, if only to what is
wrong with the ruleset.
Apart from sshd, there is practically nothing running.
atd
crond
inetd (this has nothing configured in /etc/inetd.conf, so I'll remove it).
Postfix is listening on 127.0.0.1:25 in case anything local suddenly needs to
report anything to me.
lpd (lpd was running and not listening on any ports, so I'll remove it).
Just looking for some helpful pointers on how to investigate this issue
further.
As even with a "suboptimal" rule set I wouldn't expect the box to hang.
Logs have no useful entries (certainly no "table full" messages).
Don't want to post the full ruleset here, at least not till I've been over it
with a finetoothed comb. And it is about 230 lines. On the upside quite a lot
of it has just been obseleted by us relocating all our machines to one site,
so I can shortly remove large chunks of it.
Simon
Grant. . . .
