This is simply a SYN flood attack. It may or may not be a botnet (though
saying botnet makes it sound sexier :-) ). A decent SYN flood attack tool
would randomize the source address anyway.
Some more info about the attack: All IPs were real IPs otherway the tcp
handshake wouldn't have made it. The attacker IPs were also consistent.
They also new about the blocked IPs as after a new bunch of blocked IPs
we fared OK then they added another bunch new IPs... we played this for
quite some time...
All connections were in the ESTABLISHED state.
You should try reading the following as a starting point:
http://www.securityfocus.com/infocus/1729
Your second suggestion has been implemented in the TCP/IP stack forever. The
article above gives guidance on how to tune it in a Linux implementation.
That part is about syncookies, backlog queue and half open timeouts.
None of them applies here as all connections are legitimate in terms of
SYN packets and tcp handshake.
How is the handling of ESTABLISHED connections implemented in the TCP/IP
stack?