[Firehol-support] blocklists

[Rick Marshall]

I haven't looked at this closely yet :( Been rather busy :(:(

However, I did have time to attend a security seminar headed by Trend 
Micro. Obviously this had a large promotion of their idea of security 
and how things are happening.

All good.

Question is that it seems the biggest threat now is from infected 
devices inside the network talking on their own - viruses/trojans etc to 
what Trend is calling command and control (C&C) servers that are 
directing the deep attacks.

So, my question, we have traditionally focused on external threats in to 
the organisation. Can I now turn around and track internal going out (ie 
packets originated inside the network) and going to known C&C servers 
then log/report the originating machine and the destination C&C. Would 
identify infected machines and devices - phones etc - without having to 
worry about up to date anti-virus software for some of the more serious 
threats.

I suspect this is the basis of many of the devices sold by companies 
like Trend and it is certainly a strengthening of firewalls.

This would be valuable even if I have to subscribe to a frequently 
(hourly?) updated list of C&Cs.

Please feel free to ell me this is already well covered. If not perhaps 
we could consider it.

Thanks Costa.

On 27/05/2015 8:59 am, Tsaousis, Costa wrote:
> It seems I am right.
>
> This says that the number of entries in an ipset does not affect its
> performance:
>
> http://people.netfilter.org/kadlec/nftest.pdf
>
> So, only the number of individual ipsets will make a difference and of
> course if you run blacklist(s) in 'full' or 'input'.
>
> Costa
>
>
>
> On Wed, May 27, 2015 at 1:26 AM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
>> Hi Whit,
>>
>> You are right to be concerned. I have not done any full tests on
>> performance. However this is my experience so far:
>>
>> I believe there is no problem with big ipsets. Actually the number of
>> IPs in an ipset might not be so relevant. The ipsets are sized so that
>> one lookup per packet will do the job, even if there are hundreds of
>> thousands of IPs in the ipset (I have not read the kernel source, but
>> from the ipset configuration I see that there is a hashing algorithm
>> that is sized properly to accomplish this and in a few cases when an
>> ipset is updated from the command line, the kernel logs that re-hashed
>> it). update-ipsets.sh forces a rehash by creating an new ipset and
>> swapping the active with the new during the update.
>>
>> There are also a few very simple ways to speed it up, if you are so concerned:
>>
>> 1. If you need to use 10 blacklists, create one bigger, so that
>> instead of 10 lookups, only 1 will be made. There is a tool (iprange)
>> included in the contrib dir of firehol that can aggregate ipsets,
>> giving an optimized netset result (example: cat *.ipset *.netset |
>> grep -v "^#" | ./iprange -J). This can also be done with the standard
>> aggregate tool on most distros (although it is significantly slower
>> than iprange). This is a nice feature for update-ipsets.sh too. I'll
>> add it...
>>
>> 2. Use the blacklist firehol helper with 'input' not 'full'. This will
>> only check the FIRST packet of each connection, NOT ALL the packets.
>> This limits the packets to be checked significantly.
>>
>> I have a few demanding installations:
>>
>> a. I use a few ipsets with tens of thousands of IPs each, on a
>> firewall server that has sustained traffic from 30 to 50 Mbps, with
>> one blacklist per ipset (so that I get a log with the ipset name that
>> matched it), each of them running in 'full' mode (all packets
>> checked). The overhead is ignorable. I cannot tell the difference with
>> and without the blacklists active...
>>
>> b. I have another server/gateway with 4 VDSLs and 4 ADSLs (more than
>> 250Mbps at peak) running again several ipsets in a similar fashion.
>> Again I cannot tell the difference...
>>
>> Even on my home gateway, with a simple J1900 machine (a celeron quad
>> core cpu) running ALL the ipsets in 'full' mode on my ADSL link, with
>> the link at 100% utilization (20 Mbps in total: up + down), I cannot
>> tell the difference with and without the blacklists...
>>
>> Keep in mind that I have not measured it. This is just the "feeling" I
>> have by observing 'top' and 'vmstat'...
>>
>> Costa
>>
>>
>> On Wed, May 27, 2015 at 12:09 AM, Whit Blauvelt <whit at transpect.com> wrote:
>>> Hi Costa,
>>>
>>> Since you've pushed ipsets so far, have you seen any noticable system
>>> performance issues when there are thousands, or tens of thousands, of IPs in
>>> your ipsets? On the one hand I'm thinking, "What a great idea. I'll just
>>> blacklist all the IPs on the blacklists." On the other hand, there must be
>>> some threshold where it becomes computationally expensive, and I don't have
>>> the measure of that.
>>>
>>> Thanks,
>>>
>>> Whit
> _______________________________________________
> Firehol-support mailing list
> Firehol-support at lists.firehol.org
> http://lists.firehol.org/mailman/listinfo/firehol-support

-- 
*Rick Marshall*
Director
Zenucom Pty Ltd
0411 287530
http://www.zenucom.com