[Firehol-support] FIX classid? ...and TIPS for FIREQOS_MIN_RATE_DIVISOR

[Tsaousis, Costa]

Hi Sim,

(please always CC firehol-support so that others may benefit from our
discussion - thanks)

You can map class ids to class names with this:

---

#!/bin/bash

dev="$1"
cmd="cat"

# check FireQOS names for classes
if [ -f /var/run/fireqos/ifaces/$dev ]
then
        name="`cat /var/run/fireqos/ifaces/$dev`"
        if [ ! -z "$name" -a -f /var/run/fireqos/$name.conf ]
        then
                interface_classes=
                . /var/run/fireqos/$name.conf
                opts=
                for n in $interface_classes_monitor
                do
                        classid=`echo $n | cut -d '|' -f 3`
                        classname=`echo $n | cut -d '|' -f 2`
                        opts="$opts -e 's/ $classid / $classname /g'"
                done
                test ! -z "$opts" && cmd="sed $opts"
        fi
fi

tc -s -d class show dev $dev | eval "$cmd"

---

This will give you the output of tc, but all classids will be replaced
with class names, as defined in fireqos.conf.

Now, regarding netdata:

Netdata is a highly optimized collector and database for time-series
values. The charts are just a front-end. They are handled entirely at
the browser. The server knows nothing about charting.

Check this for example:

   http://www.tsaousis.gr:19999/data/tc.world_out/100/1/average/0/0/nonzero/

It returns 100 rows of data, per second, for all classes of my
world-out fireqos interface.

This one, returns 50 rows, with average every 60 seconds:

   http://www.tsaousis.gr:19999/data/tc.world_out/50/60/average/0/0/nonzero/

This one is the same, but it gives the max value of every 60 seconds:

   http://www.tsaousis.gr:19999/data/tc.world_out/50/60/max/0/0/nonzero/

Netdata is very efficient. On my celeron J1900 processor it runs, it
only needs 7ms per second of one core for all data collection it does
per second. This is 7/1000 of a celeron. For maintaining 7200 seconds
(2 hours) history in memory for 380 time-series values, it only needs
13MB of RAM (I even wrote my own floating point number storage that
uses 29 bits for the value and 3 bits for various flags related to the
number).

Even the agregation algorithm is super fast. The 2nd link above took
5.71 ms on the server to be calculated (50x60 = 3000 values for 9
classes = 27.000 numbers averaged).

So, netdata seems already like what you propose. An efficient
real-time collector on every machine.

I have not added an authontication or access restriction mechanism to
netdata yet. For the moment if you want to remove the web front end,
just remove the web files.

Currently I plan to re-factor netdata so that:

1. Each time-series can be accessed indepedently of the others (what
you ask for)
2. Charts (i.e. collections of time-series values) can be dynamic (so
that you can say "I want a chart with the voip class of INPUT, the
voip class of OUTPUT and the sytem time of the CPU)
3. It can keep a disk database for longer periods, by storing averaged
data every 5 mins, 30 mins, 1 hour, etc.
4. Integrate events generation to it (i.e. if this number is
above/bellow this number and that number is above/bellow the other
number and these conditions are met for 10+ seconds, then fire an
event to do something).

I do not plan to create a central server. I don't think there will be
a need for one. My guess is that a web front-end with the capability
to get data from multiple netdata servers would be better. So each
server keeps its data, but a single front-end can aggregate and
combine data from all servers. This will be a lot faster and more
efficient.

On the other hand, If you really need a history of everything at a
central place, it would be more efficient to use a NOSQL db to store
the JSON output of all netdata servers, once every hour (this can be
easily automated with a script). Then the same front-end could query
the NOSQL db to get the data it needs for charting.

It is a bit different, but I think a lot better than a huge central
monitoring system.

Costa


On Sat, Jan 3, 2015 at 12:21 PM, Sim <simvirus at gmail.com> wrote:
> Hello Costa,
> thanks for your quick reply!
>
> Yes, my idea is to have one monitor server and various remote agent
> (one for server).
> In this way I can save/draw multiple charts, for multiple servers, in
> a unique page/point.
>
> For example is possibile to "call" remote agent that it will reply
> with this output command:
> /sbin/tc -s -d class show dev $interface (for example using netcat)
> Here is possible to extract all data: CLASSID - HOST - TC MIN/MAX - TC
> MIN/MAX - PRIO,
> value in/out, except the "map for class".
>
> Here a real example (with a cron/update evey 1min) and a detail for 6h
> (rrdtools chart):
> http://s4.postimg.org/zakm4ldcd/example.png
> 340 is for me the "note class", the host gp1.tout, min/max from TC and the PRIO:
>   - = 3
>  -- = 2
>   | = 4
>  + = 5
> ++ = 6 , etc...
>
> Thanks for your tip about Netdata. It's really nice! :-) I have just
> two questions:
>
> Is it possibile to "protect" access? (auth and/or IP filter)
> In "router" servers I've not iptables or other filter, and the unique way is
> using hosts.allow/deny or filter connection by service.
>
> As my example, my idea (monitor server) is to separate all charts for
> every CLASS/QoS IN+OUT entry.
> Is it possibile with Netdata?
>
> The great goals would be your Netdata client/server (agent over remote
> server, with IP source/other protection)
> and a customizable central server (where select charts to draw in a
> unique customizable dashboard)
>
> Science fiction? Sure... but it would be a good and unique tool for this :-)
>
> PS: as in other my emails... always thanks for your Firehol.... I'm
> using it from 2004!
>
> ---
> Sim
>
> 2015-01-02 19:04 GMT+01:00 Tsaousis, Costa <costa at tsaousis.gr>:
>> Hi Sim, happy new year!
>>
>> So you want to give classids yourself? Is this the fix you need?
>>
>> Keep in mind that fireqos saves a file per interface in /run/fireqos
>> that has all the maps required to get class names.
>> I have used this in netdata (https://github.com/ktsaou/netdata), a
>> tool I wrote in C for real time monitoring of linux hosts.
>>
>> Netdata demo at: http://www.tsaousis.gr:19999/
>> Scroll down to the QoS section to see FireQoS classes visualized in
>> realtime. I wrote netdata because all the graphing tools I found are
>> averaging data every 5 mins. Netdata visualizes everything per second
>> (zoom on a chart to get per second realtime moniroting), while
>> requiring very few resources!
>>
>> Regards,
>>
>> Costa
>>
>>
>> On Thu, Jan 1, 2015 at 10:47 PM, Sim <simvirus at gmail.com> wrote:
>>> Happy 2015!
>>>
>>> I'm testing FireQOS... great tool... very very thanks Costa!
>>> Before it I was using "htb" (writing the rules manually or with other tools)
>>>
>>> I've a question....
>>>
>>> Is possible to "fix" classid?
>>>
>>> It is really helpful to identify the class or building charts (with
>>> RRD for example), without losing the "index" in case of changes
>>> (add/delete before/after).
>>>
>>> PS: This is useful for distributing all the guaranteed minimum
>>> bandwidth to all classes (if not specified manually):
>>>
>>> FIREQOS_MIN_RATE_DIVISOR=`cat /etc/firehol/fireqos.conf | grep class |
>>> grep -v FIREQOS_MIN_RATE_DIVISOR | sort | uniq | wc -l`
>>>
>>> ---
>>> Sim
>>> _______________________________________________
>>> Firehol-support mailing list
>>> Firehol-support at lists.firehol.org
>>> http://lists.firehol.org/mailman/listinfo/firehol-support