3 Bandwidth monitoring tools

zabbix-dashboard-1I managed to build a vnstat web interface to plot the network traffic but the latency and the look is not the best. Also, it relies on crontabs and ssh connections, and it lacks of alarms. Now that I have a working solution,  I decided to searched a little bit more. Here you have 16 useful bandwidth monitoring tools, and my comments on some of them.

Zabbix seems to be a fully configurable dashboard, with everything you may want. The installation procedure is here. Zabbix can collect different type of data than are used to create historical graphics and output performance or load trends of the monitored targets. Suitable for small, private, and homogeneous networks, I believe.

Observium is said to supports a wide range of operating systems and hardware platforms including, Linux, Windows, FreeBSD, Cisco, HP, Dell, NetApp and so on. This is quite interesting if you want to monitor all, even the printers. The installation procedure is here . If you have a look at the demo version, you see how complicated it can become. My opinion: suitable for big, distributed, heterogeneous networks. This is not my case, so I’m not going to use it.

Cacti is the last one that I considered keeping as my final tool. It says it is used to graph time-series data of metrics such as network bandwidth utilization, CPU load, running processes, disk space etc. Therefore is like munin, but fully configurable. Here you have the installation procedure. I’m not sure RRDtool is the best one for my plots, but you may want to give it a try. I’m going to say here that cacti is suitable for small, private and heterogeneous networks.

Of course it’s a lot of work to install and configure all of them, si if you have a chance, don’t waste your time and go for the one you like more. In my case, it’s going to be Zabbix. BTW, anyone knows how the web look of its dashboard is called?

Advertisements

A vnstat web interface for CentOS 7.5

summary3We want to have an overview of the traffic on all our subnet. We could use the switch information, provided by nagios or similar, but I prefer an opensource, free, client-based solution. like vnstat, that we will use to collect the client information, and a customized web interface. To install vnstat it on CentOS 7, simply do it with yum and start the service:

yum install vnstat; systemctl start vnstat

In my case, this brought both tools we need, vnstat and vnstati.

We can try to run it already. The output a minute after it starts should be similar to this:basic-vnstat-usage

The pictures are taken from the howto in Ubuntu. Actually vnstat is the standart solution to monitor network traffic on ubuntu. There are of course more network monitoring solutions. But let’s focus on vnstat.

On the above picture, the default interface being monitored is ens33. We may need to change that on our configuration, placed on /etc/vnstat.conf.  Remember, if you do any changes, you will need to stop and start the vnstat daemon. Once we have it running saving data on a database ( named as /var/lib/vnstat/interface-name) we can periodically produce a plot with the stats we are interested in, plot that should go to our website.

You know I’m lazy. So let’s check the available options before reinventing the wheel. Option one is a simple webpage to show the vnstati images. This is not what we want. We want to collect images, but from multiple clients. It’s in addition not producing the images (summary.png, monthly.png, etc) but kudos for the layout. It’s clean and it’s easy to customize 🙂

Second option, already more complicated is vnstatsvg. I install it following the instructions. This is my output:

root@server~/vnstatsvg ## > ./configure -w /var/www/html/stats

Your Configuration:

cgi-bin directory : /var/www/cgi-bin
usr-bin directory : /usr/bin
vnstatsvg directory : /var/www/html/stats
XML dumping method: a shell script with the --dumpdb 
option provided by vnStat

checking command: vnstat... /usr/bin/vnstat
checking command: cron... which: no cron in 
(/usr/local/sbin:/...)
/usr/bin/crontab
checking command: gawk... /usr/bin/gawk
checking command: apache2... which: no apache2 in 
(/usr/local/sbin:/...)
/usr/sbin/httpd
checking command: sort... /usr/bin/sort
checking command: grep... /usr/bin/grep

Finish configuration: 
CGI_BIN: /var/www/cgi-bin/
<-- the files in cgi-bin directory of vnstatsvg will be 
installed here.
USR_BIN: /usr/bin/
<-- the vnstat binary will be installed here.
VNSTATSVG_ROOT: /var/www/html/stats/
<-- the files in admin directory of vnstatsvg will be 
installed here.
XML_DUMP_METHOD: vnstat-shell.sh
<-- the xml dumping method will be used.
The configuration have been saved in Makefile :-)

now, you can compile and install vnStatSVG as following:
$ make; make install 
root@server ~/vnstatsvg ## > make; make install
make httpclient -C src/cgi-bin/
make[1]: Entering directory `/root/vnstatsvg/src/cgi-bin'
gcc -O2 -W -o httpclient httpclient.c
make[1]: Leaving directory `/root/vnstatsvg/src/cgi-bin'
make httpclient -C src/cgi-bin/
make[1]: Entering directory `/root/vnstatsvg/src/cgi-bin'
make[1]: `httpclient' is up to date.
make[1]: Leaving directory `/root/vnstatsvg/src/cgi-bin'
Installing the administration pages...
Installing the CGI programs...
Finished installation. :-)
-----------------------------------------------

Time to go to server/stats/index.html to check out. And it works! We do get our web interface. Unfortunately, no data seems to be coming to it. Nothing even similar to this: vnstatsvg-multi-hosts

I’m not saying it’s not working. I’m going to say it’s not working for our configuration, or for our setup. I don’t know and I don’t want to dig up, since as I said, I’m lazy.  So what now? I’ll tell you what. Let’s produce the plots periodically and copy them to our webserver, to start with. Something like what I found here:

*/10 * * * * /usr/bin/vnstati -s -i eth1+eth2 -o /nfs/server_d.png

This is a crontab running each 10 minutes, saving on a png the traffic from both interfaces of my server “server” on an nfs folder /nfs/.  The idea is that all my servers will save their pictures on that folder, and I will copy also via crontab those pictures on my webserver. If I don’t have an nfs folder, I could create a crontab to ssh the pngs to my web server. As a layout, I used the two colum example from w3schools. One machine, one plot, a little bit of manual coding…and I’m done!

Could it be done better? Yes of course. I still need to try this solution with a vnstat-dashboard docker, or the dumping of the data and plotting with json or java. Why I didn’t try it? Because I know dockers, and the vnstat-dashboard will work for a single client solution, not for our multiple-client problem. If I find a proper docker solution you will be one of the first to know about it 🙂

Windows batch script tips

I’m again on fire. I want to automatize data transfers on Windows. For that, I need to write a script. Instead of going for python, this time I will try with a Windows batch script. This is just a list of tips about how to achieve certain tasks…

Open a txt file and save it as test.bat on your Desktop. You should see it now as an executable (wheels), if not, check that the extension is really bat and not txt. This will be our batch script. Now we write in the file. Let’s start with the basic. For me, it is how to comment your code (::) , and output (echo) messages. This snippet should be self explanatory:

:: This will print a message on your command prompt
echo " Executing the bat file"

If you double click on test.bat you should see a flashing black Command Prompt window appearing. If we want the prompt window to stay, we just add pause at the end of the file.

Let’s get serious. We want to ask the user to input the folder path. This means declaring a variable and asking for keyboard input. This will do it:

SET /p choice= "Please type the name of the folder :" 
:: echo '%choice%'

It could be useful to concatenate the variable with a default path ROOT.  In two lines:

SET ROOT=M:\DATA
SET SRC_ROOT=%ROOT%%choice%

We could print our new concatenated variable with echo. No need at this moment. Let’s look for a command to list all files in a folder as well as in subfolders. The line results on:

echo "Listing files in folder"
DIR /S %SRC_ROOT% /B /O:GN

We want to link the listed  files (mklink) so we have symbolic links onto another folder. In principle, what we want is to have the same tree structure but in another place. We may want that because we need to have the files “available” on a specific folder without copying them. On one line, we do it like this:

MKLINK /D C:\LinkToFolder %SRC_ROOT%

We have, of course, used our SRC_ROOT folder as imput. There are a lot of MKLINK examples on HowToGeek. We could, in principle, from the list of files, select those after a pattern, loop over the list, and link only those. But I will leave that to the reader 😀

P-hacking

p-hacking

I’m not doing heavy processing but I’m surrounded by people doing it. They have terabytes and terabytes of data that they analyze with somehow black-box tools that you can tune with multiple parameters. Then you represent the results with different programs. On an email from Graphpad, one of the companies that sell statistical analysis programs, I was asked about the complexity of my data sets and about how frequently I encounter P-Hackings on my data.

P-Hacking says that, if you try hard enough, eventually ‘statistically significant’ findings will emerge from any reasonably complicated data set.

On other words, if you know what you are looking for (for example P<0.0.5) you can end up cooking up a procedure to find it. I consider myself a bit of a data scientist and I fully agree with the possibility of inadvertently influencing the output of your analysis. I think the biggest example of this is the electoral polls. They are being ordered by governments, and nobody wants to bite the hand that gives you food…in principle. How could I avoid P-Hacking if I’m a part of the analysis procedure? I believe the solution is easier than reviewing all the analysis steps : just ask someone else to analyze your data. The problem will be then what to do if the other gets a result significantly different from yours. And that the truth is in the eye of the beholder. Or something like that 🙂

Windows robocopy configuration

I’m kind of puzzled about how different Linux rsync and Windows robocopy are. Maybe if you are a Windows user you are familiar with them, but I’m not. On StackOverflow there is a list of options to apply to robocopy, but none of them satisfy me. Here you have a comprehensive list of options. I quote:

/L :: List only - don't copy, timestamp or delete any files.
/X :: report all eXtra files, not just those selected.
/V :: produce Verbose output, showing skipped files.
/TS :: include source file Time Stamps in the output.
/FP :: include Full Pathname of files in the output.
/BYTES :: Print sizes as bytes.

/NS :: No Size - don't log file sizes.
/NC :: No Class - don't log file classes.
/NFL :: No File List - don't log file names.
/NDL :: No Directory List - don't log directory names.

/NP :: No Progress - don't display percentage copied.
/ETA :: show Estimated Time of Arrival of copied files.

/LOG:file :: output status to LOG file (overwrite existing log).
/LOG+:file :: output status to LOG file (append to existing log).

/UNILOG:file :: output status to LOG file as UNICODE 
(overwrite existing log).
/UNILOG+:file :: output status to LOG file as UNICODE 
(append to existing log).

/TEE :: output to console window, as well as the log file.

/NJH :: No Job Header.
/NJS :: No Job Summary.

/UNICODE :: output status as UNICODE.

Now what is the meaning of the output, in case you enable one? What are those “Tweaked”, “Lonely”, and “Extra” ? They refer to RoboCopy “Classes” of files:

File        Exists In   Exists In        Source/Dest     Source/Dest   Source/Dest
Class       Source      Destination      File Times      File Sizes    Attributes
=========== =========== ================ =============== ============= ============
Lonely      Yes         No               n/a             n/a           n/a
Tweaked     Yes         Yes              Equal           Equal         Different
Same        Yes         Yes              Equal           Equal         Equal
Changed     Yes         Yes              Equal           Different     n/a
Newer       Yes         Yes              Source > Dest   n/a           n/a
Older       Yes         Yes              Source < Dest   n/a           n/a
Extra       No          Yes              n/a             n/a           n/a
Mismatched  Yes (file)  Yes (directory)  n/a             n/a           n/a

The original post from Superuser is here. If you want, here it is the official Microsoft documentation, that I didn’t find very useful. I feel like I’m now the Robocopy King!

macOS and Mac annoyance list

Because not everything out there is Linux. This is a collection of solutions to problems I found while using the bitten apple products.

One important issue is always how to write files on an NTFS drive. You can read them but you can’t write them. Download the FUSE dmg file, give it access, install it and follow the instructions. It worked like a charm, but it’s a lot of work just to write on a disk. I prefer to copy the few files I’m interested via scp or some other remote copying tool like that, from the origin to the NTFS drive. And mount the drive on a computer able to write it. On this line, remember that there’s a procedure to transfer files from your mac to your Android phone using ES File Explorer. Of course, you will not be interested on it if you have an iPhone, right? But I don’t.

To install software from third parties is very tricky. Apple doesn’t want you to mess up with their carefully customised product. Here you have the how to get the security option Allow apps from anywhere on OSX. Which reminds me, it is useful to know how to recover your system using macOS Recovery. Option-Command-R is your friend. Since you want to install it again from scratch, you may get the -1006f disk error. But let’s say you manage to get a clean OSX on a new disk. Next step is to get the Applications and Accounts from your backup or external drive using the Migration Assistant. Then you can be unlucky as I was and get the External Hard Drive Error Code: -50. Be warned.

Let’s say you don’t have a mac or iPhone but you want to have, for some reason, an AppleID. Or you want to have an additional one. You can get it on the appleid web. I have several, one of them because I wanted to be an OSX developer. Really. Without leaving the subject, I had problems running Matlab on a mac, when I access via ssh to our Matlab server. This seems to be with font settings. Just type

export LC_ALL="en_US.utf8"

before starting it, and it will work. Or it worked in my case.

By the way, did you know that there are several solutions similar to yum and rpm to install programs via command line for macOS? One of them is called macports. The other is brew or homebrew. I hit this stone when trying to install FFTW for OSX. I still remember it as a not so easy thing to do. Why you want to do this, it’s not my problem.

To finish this list of annoyances, let me tell you how to flash OS images to SD cards and pen drives. You can do it with Etcher. It can be handy at one point, if you only have your poor MacBook pro with touchbar to work with.

That’s all for today, folks. I hope you enjoy your macOS as much as I do. The power of linux on a windows body 😀

C++ time operations

It’s not that I’m programming now, but I’m somehow reviewing old techniques that I don’t use since my PhD. You can play with them. To create a delay of 20 seconds:

// time delay
#include <time.h>
    time_t      startTime;
    startTime = time(NULL);
    // The loop will quit after 20 seconds
    while( time(NULL)<startTime+20 ) {
}

To measure the time since the start of the clock.

//time measurement
#include <stdio.h>
#include <time.h>
int main ()
{
  time_t seconds;
  seconds = time (NULL);
  printf ("%ld hours since January 1, 1970", seconds/3600);
  return 0;
}

Note that this is old stuff, so it may be that is no more useful. More snippets to come!