[Firehol-support] blocklists

[Tsaousis, Costa]

Oh! I forgot. The version of FireHOL on github uses iprange if it
finds it installed to reduce the number of subnets of all netsets.

Costa


On Mon, Jun 15, 2015 at 2:39 AM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
> Hi all,
>
> check this: http://ktsaou.github.io/blocklist-ipsets
>
> A lot better comparison of ipsets. Now it also maps all ipsets to see
> which countries are affected the most.
> Still updated through github every a few minutes.
>
> I have also added a few more blocklists and updated iprange to support
> all types of set theory operations (union, intersect, complement) so
> that ipsets can be manipulated easily.
>
> Costa
>
>
> On Sat, Jun 6, 2015 at 11:48 PM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
>> Hi all,
>>
>> check this:
>>
>> https://github.com/ktsaou/firehol/wiki/iprange:-optimizing-ipsets-for-iptables#reducing-ipsets
>>
>> the iprange command in firehol's contrib dir can optimize hash:net
>> ipsets to reduce the number of distinct subnets, so that the kernel
>> performs fewer lookups. The result is amazing. Most netsets I tried
>> can be expressed using 60% less subnets without any significant
>> increase to the number of entries in the set.
>>
>> I have not yet integrated it to firehol or update-ipsets.sh.
>> I have to dscuss with Phil, how this is going to be installed by the
>> standard installer before using it in the core of firehol.
>>
>> Of course, credits have to be given to Gabriel L. Somlo, the original
>> author of iprange, for creating a so efficient core upon which I built
>> all the other functionality.
>>
>> Direct link to the new source of iprange:
>> https://github.com/ktsaou/firehol/blob/master/contrib/iprange.c
>> in the same directory there is a makefile too.
>>
>> Regards,
>>
>> Costa
>>
>> On Sun, May 31, 2015 at 3:55 PM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
>>> Regarding poisoning, I contacted blocklist.de, which according to my
>>> research, seems to be the most important blocklist that could be
>>> affected by false positives, mainly because they do not control the
>>> reports they get (users are reporting the attacks they face with
>>> fail2ban).
>>>
>>> You can find the discussion I have with them here:
>>> https://forum.blocklist.de/viewtopic.php?f=4&t=244
>>>
>>> Costa
>>>
>>>
>>> On Thu, May 28, 2015 at 11:00 AM, Rick Marshall <rjm at zenucom.com> wrote:
>>>> social engineering ;)
>>>>
>>>>
>>>> On 28/05/2015 5:55 pm, Tsaousis, Costa wrote:
>>>>
>>>> An example of childish blocklist operations:
>>>>
>>>> I was searching for a VoIP blacklist and found voipbl.org.
>>>> Ok. Good. I added it to update-ipsets.sh.
>>>>
>>>> Look how these guys suggest we would update their list:
>>>>
>>>> wget -qO - http://www.voipbl.org/update/ |\
>>>>   awk '{print "if [ ! -z \""$1"\" -a \""$1"\" !=  \"#\" ]; then
>>>> /sbin/iptables -A BLACKLIST-INPUT -s \""$1"\" -j DROP;fi;"}' | sh
>>>>
>>>> or
>>>>
>>>> wget -qO - $URL -O /tmp/voipbl.txt
>>>> cat /tmp/voipbl.txt |\
>>>>   awk '{ print "if [ ! -z \""$1"\" -a \""$1"\"  != \"#\" ]; then
>>>> /usr/sbin/ipset  -A voipbl_temp \""$1"\" ;fi;"}' | sh
>>>>
>>>> I sent them an email (actually there is no way to send an email to
>>>> them - I filled a form at the company site that created voipbl.org)
>>>> saying: Hey guys, your script allows YOU to execute any command you
>>>> like as root on all systems your script runs! Example: Add to your
>>>> list this: $(echo>/tmp/voipbl-did-this). All the systems your script
>>>> runs will have the file /tmp/voipbl-did-this created as root!
>>>>
>>>> Its been days now, No response! No fix!
>>>>
>>>> Hm....
>>>>
>>>> On Thu, May 28, 2015 at 9:32 AM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
>>>>
>>>> ssh-faker is a kind of honeypot. You can contribute your logs to
>>>> blocklist.de if you like (I also added all their lists in
>>>> update-ipsets.sh yesterday, check
>>>> https://github.com/ktsaou/blocklist-ipsets - I also added a section on
>>>> this page for the lists I trust
>>>> https://github.com/ktsaou/blocklist-ipsets#which-ones-to-use).
>>>>
>>>> FireHOL v3 also supports knocks, so instead of approving IPs for ssh
>>>> access, you can enable ssh for an IP, only if you first get a connect
>>>> on port A, then B, the C within 10 seconds, in which case ssh is open
>>>> for this IP for 30 seconds.
>>>>
>>>> You can also stop port scans with FireHOL v3. You block an IP if you
>>>> get more than X attempts to connect on not-allowed different IPs or
>>>> different ports within a few seconds. This is very effective. You can
>>>> honeypot any port you like. You can honeypot IPs too. If you use
>>>> synproxy with this, the attacker will believe there is a server
>>>> listening there.
>>>>
>>>> Costa
>>>>
>>>>
>>>> On Thu, May 28, 2015 at 1:10 AM, Rick Marshall <rjm at zenucom.com> wrote:
>>>>
>>>> Thanks for that Costa. I need to think some more on this, but I agree it's
>>>> all in the lists - ingress yes, but egress seems to be the big one for
>>>> serious fraud.
>>>>
>>>> Just to update according to Trend cybercrime last year worldwide was $450b
>>>> and in the next 2-3 years is expected to reach $3t ! It's more than big
>>>> business.
>>>>
>>>> I think your most interesting point was the poisoned list problem. The way
>>>> these things work >50,000 new addresses/urls are identified every day
>>>> (mostly urls I expect) which also means a lot disappear as well. Maintaining
>>>> good lists without poisoning is difficult enough.
>>>>
>>>> We also use ssh-faker to block access to ssh in the first place - except for
>>>> approved IPs. Sort of a 2-phase access and it's been very effective.
>>>>
>>>> Still no good answer for social engineering :(
>>>>
>>>>
>>>> On 28/05/2015 1:17 am, Tsaousis, Costa wrote:
>>>>
>>>> live not leave! ok! :-)
>>>>
>>>> On Wed, May 27, 2015 at 5:02 PM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
>>>>
>>>> I have to add that at the end of the day, what you need is lists, i.e.
>>>> patterns the bad guys use:
>>>>
>>>> 1. IP blocklists to stop the bad guys at the network level
>>>> 2. Application layer blocklists to stop known application level attacks
>>>> 3. URL or domain blocklists to prevent your systems from contacting the bad
>>>> guys
>>>> 4. Intrusion detection lists that are capable of analyzing all the
>>>> layers of communication and alert you when something is wrong
>>>>
>>>> Most security companies this is what they do. They maintain such kinds
>>>> of lists. Even an anti-virus is a list of patterns and heuristics.
>>>>
>>>> Of course security companies also provide customized services, they
>>>> collect your logs to correlate the traces of the attackers and figure
>>>> out what the attackers did or trying to do. But at the end of day,
>>>> they will block them at the firewall. When you are under an attack you
>>>> may not afford to let the application level security handle it. If the
>>>> attack is massive, these layers will go out of resources quickly. So
>>>> what you need is a way to block them at the firewall.
>>>>
>>>> This is why I think ipsets are a key component (and I push it so
>>>> much). They really share key knowledge between all of us.
>>>>
>>>> Of course, there are more to be learned. What will happen if a key
>>>> blocklist most people trust is poisoned? What if somehow they have a
>>>> large number of random false positives? Do they rank dynamic IPs
>>>> properly?
>>>>
>>>> What seems missing at my view, is that the security teams around the
>>>> world have not yet found a way to exchange their knowledge is a
>>>> structured way, quickly. Each team, each company does whatever sees
>>>> fit. A few release IP lists, others have DNSBLs, others provide their
>>>> lists with a very well defined retention policy, others do not say
>>>> anything about their policies, how they remove false positives, how
>>>> they rank the attacks, etc. The InfoSec community seems very cryptic
>>>> at my eyes... or probably it is still at its very first steps.
>>>>
>>>> At the other hand, global fraud is a 30bn USD business. The fraudsters
>>>> and attackers are pros...
>>>>
>>>> Anyway, as I see it, the firewall builds walls. Blocklists are like
>>>> doors on these walls, they allow some, they block some.
>>>> Would you leave in a home without doors?
>>>>
>>>> Costa
>>>>
>>>> On Wed, May 27, 2015 at 10:24 AM, Tsaousis, Costa <costa at tsaousis.gr> wrote:
>>>>
>>>> Hi Rick.
>>>>
>>>> They are right. C&C was not just a promo. Even the attack I faced,
>>>> from the outside, was mainly originating from C&C hosts around the
>>>> world.
>>>>
>>>> Protecting a LAN of personal computers needs a different strategy
>>>> compared to protecting a LAN of servers.
>>>>
>>>> For the servers for example a basic strategy should be something like this:
>>>>
>>>> 1. A good layer 3 firewall in the front, with stateful connection
>>>> tracking, possibly with synproxy to help against SYN floods and a few
>>>> basic traps to prevent port scans. FireHOL v3+ is good at this.
>>>> synproxy support is excellent and its traps are like layer-3
>>>> honeypots.
>>>>
>>>> 2. A few blocklists to keep the well known bad guys out of the
>>>> services your firewall allows to pass. FireHOL v3+ is also good at
>>>> this. Of course, the blocklists will not help you much if you get a
>>>> zero-day attack (you are first to be attacked on the net).
>>>>
>>>> 3. An application level security layer (e.g. modsecurity for http).
>>>> This will detect known attacks at the application layer (sql
>>>> injection, known application vulnerabilities, etc). This also works
>>>> with lists (pattern matching rules) that should be updated frequently.
>>>>
>>>> 4. fail2ban on all services where possible (you should even contribute
>>>> to the blocklists with this). This will protect you against brute
>>>> force attacks.
>>>>
>>>> 5. all outgoing http/https requests the servers make should go through
>>>> a proxy, so that you can monitor what your servers are requesting from
>>>> the internet and possibly add some security there (url blocklists -
>>>> the 3rd set of lists you will need, anti-malware and anti-virus
>>>> checks). Also, tt the firewall level, you should not have "client all
>>>> accept" for servers. You should allow very specific things, probably
>>>> with 'src' and 'dst' on every client rule.
>>>>
>>>> Some guys also add a snort just behind the firewall that sniffs all
>>>> traffic and logs anomalies. In this case you will need a 4th type of
>>>> lists, the snort alert rules. If you do this, then most probably you
>>>> should add an ssl accelerator somewhere, so that it can read https
>>>> traffic (this is also required if you add modsecurity).
>>>>
>>>> You can also contribute to the overall internet security by installing
>>>> application honeyports - this will be a contribution to blocklists.
>>>>
>>>> Finally, you can design your application to handle security
>>>> information, such as IP reputation data. Project Honey Pot provides
>>>> such a mechanism through DNS (httpbl.org).
>>>>
>>>> On a LAN of personal computers, things are different. In this case the
>>>> blocklists will help you after you are infected (there are a few lists
>>>> that will block viruses, like zeus, feodo, pelevo, etc), but generally
>>>> your public IPs will appear on the blocklists... On LAN PCs a proxy
>>>> with anti-virus and anti-malware is required and a good anti-virus
>>>> anti-malware is required on every PC, especially Windows. A snort
>>>> installation can also help to detect anomalies on a LAN of PCs.
>>>>
>>>> Costa
>>>>
>>>>
>>>> On Wed, May 27, 2015 at 2:51 AM, Rick Marshall <rjm at zenucom.com> wrote:
>>>>
>>>> I haven't looked at this closely yet :( Been rather busy :(:(
>>>>
>>>> However, I did have time to attend a security seminar headed by Trend Micro.
>>>> Obviously this had a large promotion of their idea of security and how
>>>> things are happening.
>>>>
>>>> All good.
>>>>
>>>> Question is that it seems the biggest threat now is from infected devices
>>>> inside the network talking on their own - viruses/trojans etc to what Trend
>>>> is calling command and control (C&C) servers that are directing the deep
>>>> attacks.
>>>>
>>>> So, my question, we have traditionally focused on external threats in to the
>>>> organisation. Can I now turn around and track internal going out (ie packets
>>>> originated inside the network) and going to known C&C servers then
>>>> log/report the originating machine and the destination C&C. Would identify
>>>> infected machines and devices - phones etc - without having to worry about
>>>> up to date anti-virus software for some of the more serious threats.
>>>>
>>>> I suspect this is the basis of many of the devices sold by companies like
>>>> Trend and it is certainly a strengthening of firewalls.
>>>>
>>>> This would be valuable even if I have to subscribe to a frequently (hourly?)
>>>> updated list of C&Cs.
>>>>
>>>> Please feel free to ell me this is already well covered. If not perhaps we
>>>> could consider it.
>>>>
>>>> Thanks Costa.
>>>>
>>>>
>>>> On 27/05/2015 8:59 am, Tsaousis, Costa wrote:
>>>>
>>>> It seems I am right.
>>>>
>>>> This says that the number of entries in an ipset does not affect its
>>>> performance:
>>>>
>>>> http://people.netfilter.org/kadlec/nftest.pdf
>>>>
>>>> So, only the number of individual ipsets will make a difference and of
>>>> course if you run blacklist(s) in 'full' or 'input'.
>>>>
>>>> Costa
>>>>
>>>>
>>>>
>>>> On Wed, May 27, 2015 at 1:26 AM, Tsaousis, Costa <costa at tsaousis.gr>
>>>> wrote:
>>>>
>>>> Hi Whit,
>>>>
>>>> You are right to be concerned. I have not done any full tests on
>>>> performance. However this is my experience so far:
>>>>
>>>> I believe there is no problem with big ipsets. Actually the number of
>>>> IPs in an ipset might not be so relevant. The ipsets are sized so that
>>>> one lookup per packet will do the job, even if there are hundreds of
>>>> thousands of IPs in the ipset (I have not read the kernel source, but
>>>> from the ipset configuration I see that there is a hashing algorithm
>>>> that is sized properly to accomplish this and in a few cases when an
>>>> ipset is updated from the command line, the kernel logs that re-hashed
>>>> it). update-ipsets.sh forces a rehash by creating an new ipset and
>>>> swapping the active with the new during the update.
>>>>
>>>> There are also a few very simple ways to speed it up, if you are so
>>>> concerned:
>>>>
>>>> 1. If you need to use 10 blacklists, create one bigger, so that
>>>> instead of 10 lookups, only 1 will be made. There is a tool (iprange)
>>>> included in the contrib dir of firehol that can aggregate ipsets,
>>>> giving an optimized netset result (example: cat *.ipset *.netset |
>>>> grep -v "^#" | ./iprange -J). This can also be done with the standard
>>>> aggregate tool on most distros (although it is significantly slower
>>>> than iprange). This is a nice feature for update-ipsets.sh too. I'll
>>>> add it...
>>>>
>>>> 2. Use the blacklist firehol helper with 'input' not 'full'. This will
>>>> only check the FIRST packet of each connection, NOT ALL the packets.
>>>> This limits the packets to be checked significantly.
>>>>
>>>> I have a few demanding installations:
>>>>
>>>> a. I use a few ipsets with tens of thousands of IPs each, on a
>>>> firewall server that has sustained traffic from 30 to 50 Mbps, with
>>>> one blacklist per ipset (so that I get a log with the ipset name that
>>>> matched it), each of them running in 'full' mode (all packets
>>>> checked). The overhead is ignorable. I cannot tell the difference with
>>>> and without the blacklists active...
>>>>
>>>> b. I have another server/gateway with 4 VDSLs and 4 ADSLs (more than
>>>> 250Mbps at peak) running again several ipsets in a similar fashion.
>>>> Again I cannot tell the difference...
>>>>
>>>> Even on my home gateway, with a simple J1900 machine (a celeron quad
>>>> core cpu) running ALL the ipsets in 'full' mode on my ADSL link, with
>>>> the link at 100% utilization (20 Mbps in total: up + down), I cannot
>>>> tell the difference with and without the blacklists...
>>>>
>>>> Keep in mind that I have not measured it. This is just the "feeling" I
>>>> have by observing 'top' and 'vmstat'...
>>>>
>>>> Costa
>>>>
>>>>
>>>> On Wed, May 27, 2015 at 12:09 AM, Whit Blauvelt <whit at transpect.com>
>>>> wrote:
>>>>
>>>> Hi Costa,
>>>>
>>>> Since you've pushed ipsets so far, have you seen any noticable system
>>>> performance issues when there are thousands, or tens of thousands, of
>>>> IPs in
>>>> your ipsets? On the one hand I'm thinking, "What a great idea. I'll just
>>>> blacklist all the IPs on the blacklists." On the other hand, there must
>>>> be
>>>> some threshold where it becomes computationally expensive, and I don't
>>>> have
>>>> the measure of that.
>>>>
>>>> Thanks,
>>>>
>>>> Whit
>>>>
>>>> _______________________________________________
>>>> Firehol-support mailing list
>>>> Firehol-support at lists.firehol.org
>>>> http://lists.firehol.org/mailman/listinfo/firehol-support
>>>>
>>>> --
>>>> *Rick Marshall*
>>>> Director
>>>> Zenucom Pty Ltd
>>>> 0411 287530
>>>> http://www.zenucom.com
>>>> _______________________________________________
>>>> Firehol-support mailing list
>>>> Firehol-support at lists.firehol.org
>>>> http://lists.firehol.org/mailman/listinfo/firehol-support
>>>>
>>>>
>>>> --
>>>> Rick Marshall
>>>> Director
>>>> Zenucom Pty Ltd
>>>> 0411 287530
>>>> http://www.zenucom.com
>>>>
>>>>
>>>> --
>>>> Rick Marshall
>>>> Director
>>>> Zenucom Pty Ltd
>>>> 0411 287530
>>>> http://www.zenucom.com