[Firehol-support] better understanding link-balancer and PBR

[Tsaousis, Costa]

> you mentioned this thing about CONNMARK and FW, could you elaborate on
that? I was just considering what's gonna happen to the connections who
have already been established and MASQUERADED to go out of a certain
link/with a certain ip.

Well, when NAT or CONNMARK are involved and a gateway goes down, the
connections using this gateway will continue to be in the connection
tracker table (conntrack -L). There is no way to save these connections.
They have to be re-established. This will of course happen by itself, but
depending on the setup and the kind of traffic (e.g. UDP), a timeout may be
involved.

To avoid this timeout, I usually run conntrack -D ... to delete the
problematic entries from the connection tracker table. Once they are
deleted, the next packet seen on these connections by the connection
tracker will be marked as INVALID, which depending on the firewall
configuration, might trigger an ICMP Host Unreachable or something similar
that will let the sender know the connection is broken.

To handle this change you can override the BASH function
`updated_routes()`:
https://github.com/firehol/firehol/blob/183434247090603fa8580a489c3f50d3fef1f75b/sbin/link-balancer#L1194-L1201

Link-balancer calls this function every time a routing table is updated:
https://github.com/firehol/firehol/blob/183434247090603fa8580a489c3f50d3fef1f75b/sbin/link-balancer#L1121

To check which entries you need to delete, you will have to test it.
Usually TCP connections need nothing. Long living UDP sockets might need
this.

Costa


On Thu, Dec 8, 2016 at 7:23 PM, Spike <spike at drba.org> wrote:

> inline below.c
>
> On Wed, Dec 7, 2016 at 11:16 AM Tsaousis, Costa <costa at tsaousis.gr> wrote:
> [snip]
>
>>
>> So, link-balancer copies the static routes, to provide a seamless
>> experience for your static routes. You only need to care about the default
>> gateway in policy based routing. All static routes just work.
>>
>
> ah, I get it now, thank you for your patience and clear explanation.
>
> maybe I didn't ask this clearly, lemme try again. I'm wondering if when
>> the kernel chooses the default nexthop route in main that triggers another
>> pass of the rules or not. Does that make more sense?
>>
>>
>> no. policy based routing (ip rules) is applied prior to the routing
>> decision. The routing decision is the routing table itself. So a packet is
>> to be forwarded, then the policy rules are consulted to decide the routing
>> table (this is called policy based routing), then a lookup is made on the
>> routing table to decide where to send the packet to.
>>
>
> ok, thanks
>
>
>> hm... I don't know how the routing cache works exactly. I know however,
>> that in all cases I have encountered so far, my problem was only the
>> iptables connection tracker, especially when NAT is involved or CONNMARK is
>> used.
>> I had to to run conntrack to delete all the rules of the failed gateway,
>> to prevent long timeouts.
>>
>>
> you mentioned this thing about CONNMARK and FW, could you elaborate on
> that? I was just considering what's gonna happen to the connections who
> have already been established and MASQUERADED to go out of a certain
> link/with a certain ip.
>
>
>> The whole idea is to keep policy based routing as stable as possible. So,
>> link-balancer will never change policy based rules based on the state of
>> the gateways. The only change when gateways change state are the default
>> gateway of one or more routing tables. It does this optimally. For example,
>> if you had PPPoE connections, when a PPPoE link goes down, only the default
>> gateway of the tables this gateway is used, is affected (the kernel removes
>> the default gateway automatically by itself). So, link-balancer just
>> restores a gateway on these tables. Nothing more.
>>
>
> got it, thank you
>
>
>> The ping-pong is a logical problem. You are using servers on the internet
>> to test the gateways. Just make sure these servers are only accessed via
>> the same gateway, independently of the state of the gateways. If you
>> achieve this, there will no ping-pong. This is also why the RAS of your
>> ISPs are probably a better choice. If you have multiple ISPs, you will most
>> probably not be able to ping the RAS of ISP A via ISP B.
>>
>
> Thanks for this, you were right, once I setup the static route from the
> other thread to solve the problem of no initial routes that also addressed
> this problem.
>
> thanks again,
>
> Spike
>