[Firehol-support] blocklists

[Tsaousis, Costa]

Hi Whit,

You are right to be concerned. I have not done any full tests on
performance. However this is my experience so far:

I believe there is no problem with big ipsets. Actually the number of
IPs in an ipset might not be so relevant. The ipsets are sized so that
one lookup per packet will do the job, even if there are hundreds of
thousands of IPs in the ipset (I have not read the kernel source, but
from the ipset configuration I see that there is a hashing algorithm
that is sized properly to accomplish this and in a few cases when an
ipset is updated from the command line, the kernel logs that re-hashed
it). update-ipsets.sh forces a rehash by creating an new ipset and
swapping the active with the new during the update.

There are also a few very simple ways to speed it up, if you are so concerned:

1. If you need to use 10 blacklists, create one bigger, so that
instead of 10 lookups, only 1 will be made. There is a tool (iprange)
included in the contrib dir of firehol that can aggregate ipsets,
giving an optimized netset result (example: cat *.ipset *.netset |
grep -v "^#" | ./iprange -J). This can also be done with the standard
aggregate tool on most distros (although it is significantly slower
than iprange). This is a nice feature for update-ipsets.sh too. I'll
add it...

2. Use the blacklist firehol helper with 'input' not 'full'. This will
only check the FIRST packet of each connection, NOT ALL the packets.
This limits the packets to be checked significantly.

I have a few demanding installations:

a. I use a few ipsets with tens of thousands of IPs each, on a
firewall server that has sustained traffic from 30 to 50 Mbps, with
one blacklist per ipset (so that I get a log with the ipset name that
matched it), each of them running in 'full' mode (all packets
checked). The overhead is ignorable. I cannot tell the difference with
and without the blacklists active...

b. I have another server/gateway with 4 VDSLs and 4 ADSLs (more than
250Mbps at peak) running again several ipsets in a similar fashion.
Again I cannot tell the difference...

Even on my home gateway, with a simple J1900 machine (a celeron quad
core cpu) running ALL the ipsets in 'full' mode on my ADSL link, with
the link at 100% utilization (20 Mbps in total: up + down), I cannot
tell the difference with and without the blacklists...

Keep in mind that I have not measured it. This is just the "feeling" I
have by observing 'top' and 'vmstat'...

Costa


On Wed, May 27, 2015 at 12:09 AM, Whit Blauvelt <whit at transpect.com> wrote:
> Hi Costa,
>
> Since you've pushed ipsets so far, have you seen any noticable system
> performance issues when there are thousands, or tens of thousands, of IPs in
> your ipsets? On the one hand I'm thinking, "What a great idea. I'll just
> blacklist all the IPs on the blacklists." On the other hand, there must be
> some threshold where it becomes computationally expensive, and I don't have
> the measure of that.
>
> Thanks,
>
> Whit