and I admit it could be easily tested if "--set" updates an already existing "--set" or not, but I have to leave right now. I think I'll examine that when I'm back if nobody is faster ;-) )
well in fact I just run a test to see if it's true, and I had already writen a long post to ask why is it so, as the ruleset with --hitcount 3 dropped normally the third connection (it doesn't have to be set to 6),
and then it hit me (--hitcount 1 ;) that the --update is AFTER the "-j RETURN"
and it runs ok now, the --update rule can be easily removed, as the --set does the trick the way it should.
My ruleset is now something like this below, and works as I intended it to, 2 connections ok, third one drops.
ipt=/usr/local/sbin/iptables
$ipt -N SSH_Brute_Force
$ipt -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
$ipt -A INPUT -s ! $MyIP -m recent --rcheck --name SSH -j SSH_Brute_Force
$ipt -A SSH_Brute_Force -m recent ! --rcheck --name SSH --seconds 60 --hitcount 3 -j RETURN
$ipt -A SSH_Brute_Force -j LOG --log-prefix "SSH Brute Force Attempt: "
$ipt -A SSH_Brute_Force -j DROP
# do any other checks on the returned ssh packets
$ipt -A INPUT -m recent --rcheck --name SSH -j ACCEPT
-- With regards Łukasz Hejnak