Yes firewalls do suck, but one of the reasons they suck a lot worse than they need to is because there was a lot of resistance in the IETF to the whole concept. And so any attempt to make IETF protocols firewall friendly was often met with obstructionism.
One long term consequence of this obstructionism is that nobody actually deploys what IETF claims is the IPSEC standard. Microsoft and others implement but every company I have been at with a VPN has required use of a plug-in to get round the intentional NAT-sabotage etc.
The question that seems to never be asked but should is how we make firewalls less of a problem.
The first step as I see it is to acknowledge the fact that nobody has any right to send any packets into my network unless I ask for them.
The end to end principle is a good one. The end-to-end ideology is for idiots who should actually read the paper rather than preaching it like it is their little red book. Anyone who does not understand the difference between a network and an inter-network needs to collect their plate and clock.
I come from the process control world. We don't like noisy networks. It should be possible to slap a network analyzer on any wire at any time and be able to understand all the traffic flowing. So I really don't want floods of external packets coming from outside my network.
For me there is a big difference between the network I am responsible for and everything else.
We had the same problem in the email-spam community. Even today some people just can't get the fact that just because you want to send me a message does not mean I have to provide resources to accept it.
Outbound traffic is relatively easy to deal with. All the firewall needs to do is to decide whether the destination is one that isn't permitted. And usually the right decision gets made - though there are many enterprise firewalls locked down to only permit outbound port 80 and 443 and nothing else unless the packets come from a specially privileged server.OK this is bad but at least the firewall logs tell us the extent of the issue.
The problem comes with the inbound packets, should they be allowed through or not. And here there is a signaling gap because in the Berkley sockets stack there is no distinction between a network server and an Internetwork server. Opening a socket only requires the necessary privileges on the host. There is no concept of asking for socket to be created that is to be visible on the external Internet because the distinction between the network and the Internetwork is not made. And of course it didn't really exist in the days when computers cost $100K and up.
At the moment a firewall can't do the right thing because it does not have the right information. Giving it the right information is a necessary but not sufficient condition to doing the right thing.
This is one of the functions I support in Omnibroker. When an application wants to open an inbound or outbound network connection it makes a request to the Omnibroker which then performs the necessary configuration and supplies all the necessary information to make the service connection.
In the case of a client connection this would be the IP address and port to connect to and the transport (TCP/TLS/UDP/ NextP) and application protocols (IMAPvsPOP3 for example) to use.
In the case of a server connection this would be telling the server what the external IP address and port are and possibly include TLS credentials (for cloud services often the private key).
In either case the difference is that if the operation fails because the local policy does not permit it, the application is told that this is the reason and can tell the user. Current generation firewalls are unsatisfactory because the end user can't distinguish between an operation that fails due to policy and an operation that fails due to the network being bjorked.
Note that this is moving beyond firewalls. Firewalls are a weak security solution because they only provide policy enforcement at the perimeter. In a defense in depth strategy we would want every device in the network to perform policy enforcement and policy audit.
No device should pass a packet that is not explicitly authorized by network policy. Every packet that violates the network policy should cause an alert.
This is of course an approach that is currently impractical because we don't have pervasive network information and so can't determine what the policy should be and tell devices what to enforce. But moving to the omnibroker model makes the tight security model practical.