[Firehol-devs] code generation

[Tsaousis, Costa]

Nice.
I'll do it like this then...

Thanks Phil.


On Thu, Feb 12, 2015 at 10:03 PM, Phil Whineray <phil at sanewall.org> wrote:
> Hi Costa
>
> I think the current ESTABLISHED + RELATED rule only matches e.g. ftp data
> connections and other things that are tracked specially. This rule
> does not match normal established traffic.
>
> For your proposal we will need 2 rules, for ESTABISHED and RELATED
> separately, so that the majority of traffic matches early.
>
> However I would sooner see this as an option e.g.
>   FIREHOL_RULESET=optimal
> v.s.
>   FIREHOL_RULESET=complete
>
> I can see that many people might prefer the more efficient and shorter
> firewall. For me at least, though, the current behaviour is worth the extra
> rules. I very much appreciate that traffic is checked in both directions
> and pre-established connections will be severed if I update my firewall
> to deny them.
>
> I am assuming that since the change mostly just stops the generation of
> reverse rules, it should not be too hard to put these behind a test?
>
> In optimal mode we could just re-enable the reverse behaviour automatically
> and temporarily when the user creates a stateless rule (and add the
> prerouting rules). If the user tries using accounting in this mode,
> then move the ESTABLISHED accept to the end and warn the user the
> firewall is no longer fully optimal.
>
> Cheers
> Phil
>
> On Thu, Feb 12, 2015 at 02:24:11PM +0200, Tsaousis, Costa wrote:
>> There is another major change I would like to consider:
>>
>> Today, FireHOL generates stateful rules for both request and replies, like this:
>>
>> rule in ... state NEW,ESTABLISHED ... action ACCEPT
>> rule out ... state ESTABLISHED ... action ACCEPT
>>
>> and at the end of the firewall there is a rule that accepts all
>> ESTABLISHED and RELATED traffic.
>>
>> I think the optimal should be:
>>
>> 1. accept ESTABLISHED and RELATED traffic at the top of the firewall
>> (instead of the bottom); given that most of the traffic is replies,
>> this will significantly reduce the iptables rules the packets have to
>> traverse in order to be accepted.
>>
>> 2. each stateful rule generates code only for state NEW packets (the
>> requests), taking of course into account client ports (as it already
>> does);  this will cut half of the rules generated today (no need for
>> matching per statement ESTABLISHED traffic).
>>
>> It will also make certain features (such as logging and ipset counters
>> update) be executed only for requests.
>>
>> At the same time, it will allow us to incorporate into firehol new and
>> exciting features like SYNPROXY (check
>> http://people.netfilter.org/hawk/presentations/devconf2014/iptables-ddos-mitigation_JesperBrouer.pdf
>> and https://r00t-services.net/knowledgebase/14/Homemade-DDoS-Protection-Using-IPTables-SYNPROXY.html)
>>
>> Unfortunately however, this will break:
>>
>> a. accounting, since we will be unable to track accounting for the
>> replies; the same will happen for ipset counters updates that the user
>> wants to update for the whole traffic.
>>
>> Both of these features are relatively new. My understanding is that
>> optimal iptables rules generation should be of higher priority. We
>> could just say, this is the way these features work in firehol and
>> possibly create a few helpers that will allow the users do these
>> functions the way they want.
>>
>> b. stateless rules. I believe this is faulty already. stateless rules
>> should have disabled connection tracking for their traffic at the raw
>> table and then apply their rules outside the stateful filtering rules.
>> We could have a helper to do it right like this:
>>
>> stateless server smtp accept inface eth0 ...
>>
>> The above should be given before all interfaces and routers. It is a
>> helper and should generate rules before the stateful packet filtering.
>> We could allow 'group with' work also outside the interfaces and
>> routers, like this:
>>
>> group with inface eth0
>>    stateless server smtp accept
>>    stateless server http accept
>> group end
>>
>> interface eth0 internet
>>    server imapd accept
>>    ...
>>
>> What do you say?
>> Is this the firehol we want?
>> Do you find any issues I haven't looked at?
>>
>> Costa
>>
>>
>> On Thu, Feb 12, 2015 at 2:17 AM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
>> > Hi all, I made a few more changes in rule().
>> >
>> > I have commented the code to let you know what I have done.
>> >
>> > The improvement is significant in certain cases. I optimized branching
>> > (now it is avoided unless it is absolutely necessary to have it), made
>> > certain feature (such as limit and ipset) be executed only once per
>> > rule, etc. As I see it, rule() cannot be made to generate better
>> > iptables code. It is optimal now.
>> >
>> > I think I have tested it a lot.
>> > If you can test it, please let me know if you find any issues.
>> >
>> > Overall, FireHOL needs another optimization when grouping multiple
>> > services together, like this:
>> >
>> > server "smtp ftp http ssh" accept src "10.0.0.0/8" log "My Servers"
>> >
>> > This should internally be interpreted as:
>> >
>> > group with src "10.0.0.0/8" log "My Servers"
>> >       server "smtp ftp http ssh" accept
>> > group end
>> >
>> > The above will provide optimal iptables code.
>> >
>> > PS: the new commit is 10-20% slower. check the code. it does a lot more.
>> >
>> > Examples:
>> >
>> > Command  : server smtp accept src "1.1.1.1 2.2.2.2 3.3.3.3" log "SMTP TRAFFIC"
>> >
>> > OLD
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 1.1.1.1 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 1.1.1.1 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > ACCEPT
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 2.2.2.2 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 2.2.2.2 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > ACCEPT
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 3.3.3.3 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 3.3.3.3 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 2.2.2.2 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 2.2.2.2 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 3.3.3.3 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 3.3.3.3 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > ACCEPT
>> >
>> > NEW
>> > /sbin/iptables -t filter -N in_world_smtp_s1.1
>> > /sbin/iptables -t filter -A in_world_smtp_s1.1 -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s1.1 -j ACCEPT
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 1.1.1.1 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > in_world_smtp_s1.1
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 2.2.2.2 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > in_world_smtp_s1.1
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 3.3.3.3 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > in_world_smtp_s1.1
>> > /sbin/iptables -t filter -N out_world_smtp_s1.2
>> > /sbin/iptables -t filter -A out_world_smtp_s1.2 -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s1.2 -j ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > out_world_smtp_s1.2
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 2.2.2.2 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > out_world_smtp_s1.2
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 3.3.3.3 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > out_world_smtp_s1.2
>> >
>> >
>> > ---
>> >
>> > Command  : server smtp accept src not 1.1.1.1 dst not 2.2.2.2 log "HELLO"
>> >
>> > OLD
>> > /sbin/iptables -t filter -N in_world_smtp_s3.1
>> > /sbin/iptables -t filter -A in_world_smtp_s3.1 -s 1.1.1.1 -j RETURN
>> > /sbin/iptables -t filter -A in_world_smtp_s3.1 -d 2.2.2.2 -j RETURN
>> > /sbin/iptables -t filter -A in_world_smtp_s3.1 -j NFLOG --nflog-prefix=HELLO:
>> > /sbin/iptables -t filter -A in_world_smtp_s3.1 -p tcp -j ACCEPT
>> > /sbin/iptables -t filter -A in_world_smtp_s3 -p tcp --sport 1024:65535
>> > --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > in_world_smtp_s3.1
>> > /sbin/iptables -t filter -N out_world_smtp_s3.2
>> > /sbin/iptables -t filter -A out_world_smtp_s3.2 -s 2.2.2.2 -j RETURN
>> > /sbin/iptables -t filter -A out_world_smtp_s3.2 -d 1.1.1.1 -j RETURN
>> > /sbin/iptables -t filter -A out_world_smtp_s3.2 -j NFLOG --nflog-prefix=HELLO:
>> > /sbin/iptables -t filter -A out_world_smtp_s3.2 -p tcp -j ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s3 -p tcp --sport 25
>> > --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > out_world_smtp_s3.2
>> >
>> > NEW
>> > /sbin/iptables -t filter -A in_world_smtp_s2 -p tcp \! -s 1.1.1.1
>> > --sport 1024:65535 \! -d 2.2.2.2 --dport 25 -m conntrack --ctstate
>> > NEW\,ESTABLISHED -j NFLOG --nflog-prefix=HELLO:
>> > /sbin/iptables -t filter -A in_world_smtp_s2 -p tcp \! -s 1.1.1.1
>> > --sport 1024:65535 \! -d 2.2.2.2 --dport 25 -m conntrack --ctstate
>> > NEW\,ESTABLISHED -j ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s2 -p tcp \! -s 2.2.2.2
>> > --sport 25 \! -d 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate
>> > ESTABLISHED -j NFLOG --nflog-prefix=HELLO:
>> > /sbin/iptables -t filter -A out_world_smtp_s2 -p tcp \! -s 2.2.2.2
>> > --sport 25 \! -d 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate
>> > ESTABLISHED -j ACCEPT
>> >
>> >
>> >
>> > Command  : server smtp accept src "1.1.1.1 2.2.2.2 3.3.3.3" ipset trap
>> > src log "SMTP TRAFFIC"
>> >
>> > OLD
>> > # Rules for smtp server, with server port(s) 'tcp/25' and client
>> > port(s) 'default'
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 1.1.1.1 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -m set
>> > --match-set trap src -j NFLOG --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 1.1.1.1 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -m set
>> > --match-set trap src -j ACCEPT
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 2.2.2.2 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -m set
>> > --match-set trap src -j NFLOG --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 2.2.2.2 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -m set
>> > --match-set trap src -j ACCEPT
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 3.3.3.3 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -m set
>> > --match-set trap src -j NFLOG --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s1 -p tcp -s 3.3.3.3 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -m set
>> > --match-set trap src -j ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -m set
>> > --match-set trap src -j NFLOG --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -m set
>> > --match-set trap src -j ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 2.2.2.2 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -m set
>> > --match-set trap src -j NFLOG --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 2.2.2.2 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -m set
>> > --match-set trap src -j ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 3.3.3.3 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -m set
>> > --match-set trap src -j NFLOG --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s1 -p tcp --sport 25 -d
>> > 3.3.3.3 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -m set
>> > --match-set trap src -j ACCEPT
>> >
>> > NEW - check that the ipset command that updates ipset counters is
>> > executed only once
>> > /sbin/iptables -t filter -N in_world_smtp_s3.3
>> > /sbin/iptables -t filter -A in_world_smtp_s3.3 -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A in_world_smtp_s3.3 -m set --match-set trap
>> > src -j ACCEPT
>> > /sbin/iptables -t filter -A in_world_smtp_s3 -p tcp -s 1.1.1.1 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > in_world_smtp_s3.3
>> > /sbin/iptables -t filter -A in_world_smtp_s3 -p tcp -s 2.2.2.2 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > in_world_smtp_s3.3
>> > /sbin/iptables -t filter -A in_world_smtp_s3 -p tcp -s 3.3.3.3 --sport
>> > 1024:65535 --dport 25 -m conntrack --ctstate NEW\,ESTABLISHED -j
>> > in_world_smtp_s3.3
>> > /sbin/iptables -t filter -N out_world_smtp_s3.4
>> > /sbin/iptables -t filter -A out_world_smtp_s3.4 -j NFLOG
>> > --nflog-prefix=SMTP_TRAFFIC:
>> > /sbin/iptables -t filter -A out_world_smtp_s3.4 -m set --match-set
>> > trap src -j ACCEPT
>> > /sbin/iptables -t filter -A out_world_smtp_s3 -p tcp --sport 25 -d
>> > 1.1.1.1 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > out_world_smtp_s3.4
>> > /sbin/iptables -t filter -A out_world_smtp_s3 -p tcp --sport 25 -d
>> > 2.2.2.2 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > out_world_smtp_s3.4
>> > /sbin/iptables -t filter -A out_world_smtp_s3 -p tcp --sport 25 -d
>> > 3.3.3.3 --dport 1024:65535 -m conntrack --ctstate ESTABLISHED -j
>> > out_world_smtp_s3.4