[Firehol-support] blocklists

[Tsaousis, Costa]

It seems I am right.

This says that the number of entries in an ipset does not affect its
performance:

http://people.netfilter.org/kadlec/nftest.pdf

So, only the number of individual ipsets will make a difference and of
course if you run blacklist(s) in 'full' or 'input'.

Costa



On Wed, May 27, 2015 at 1:26 AM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
> Hi Whit,
>
> You are right to be concerned. I have not done any full tests on
> performance. However this is my experience so far:
>
> I believe there is no problem with big ipsets. Actually the number of
> IPs in an ipset might not be so relevant. The ipsets are sized so that
> one lookup per packet will do the job, even if there are hundreds of
> thousands of IPs in the ipset (I have not read the kernel source, but
> from the ipset configuration I see that there is a hashing algorithm
> that is sized properly to accomplish this and in a few cases when an
> ipset is updated from the command line, the kernel logs that re-hashed
> it). update-ipsets.sh forces a rehash by creating an new ipset and
> swapping the active with the new during the update.
>
> There are also a few very simple ways to speed it up, if you are so concerned:
>
> 1. If you need to use 10 blacklists, create one bigger, so that
> instead of 10 lookups, only 1 will be made. There is a tool (iprange)
> included in the contrib dir of firehol that can aggregate ipsets,
> giving an optimized netset result (example: cat *.ipset *.netset |
> grep -v "^#" | ./iprange -J). This can also be done with the standard
> aggregate tool on most distros (although it is significantly slower
> than iprange). This is a nice feature for update-ipsets.sh too. I'll
> add it...
>
> 2. Use the blacklist firehol helper with 'input' not 'full'. This will
> only check the FIRST packet of each connection, NOT ALL the packets.
> This limits the packets to be checked significantly.
>
> I have a few demanding installations:
>
> a. I use a few ipsets with tens of thousands of IPs each, on a
> firewall server that has sustained traffic from 30 to 50 Mbps, with
> one blacklist per ipset (so that I get a log with the ipset name that
> matched it), each of them running in 'full' mode (all packets
> checked). The overhead is ignorable. I cannot tell the difference with
> and without the blacklists active...
>
> b. I have another server/gateway with 4 VDSLs and 4 ADSLs (more than
> 250Mbps at peak) running again several ipsets in a similar fashion.
> Again I cannot tell the difference...
>
> Even on my home gateway, with a simple J1900 machine (a celeron quad
> core cpu) running ALL the ipsets in 'full' mode on my ADSL link, with
> the link at 100% utilization (20 Mbps in total: up + down), I cannot
> tell the difference with and without the blacklists...
>
> Keep in mind that I have not measured it. This is just the "feeling" I
> have by observing 'top' and 'vmstat'...
>
> Costa
>
>
> On Wed, May 27, 2015 at 12:09 AM, Whit Blauvelt <whit at transpect.com> wrote:
>> Hi Costa,
>>
>> Since you've pushed ipsets so far, have you seen any noticable system
>> performance issues when there are thousands, or tens of thousands, of IPs in
>> your ipsets? On the one hand I'm thinking, "What a great idea. I'll just
>> blacklist all the IPs on the blacklists." On the other hand, there must be
>> some threshold where it becomes computationally expensive, and I don't have
>> the measure of that.
>>
>> Thanks,
>>
>> Whit