Filtering CSV data with awk

There are command line tools that can do a very good job filtering your data. Let’s say on a big CSV file you have an unordered list of names and values on this way:


It can be the result of the merging of several files. I will call the file values.csv. The names can be repeated or not: it doesn’t matter for this example. Also some numbers can be missing, no problem. But now that we have a lot of information onto our file, we want to filter it. Let’s say we don’t want any basil on our record, and only people with the second value greater than 40. We can do that with awk and grep. Like this:

grep -v basil values.csv | awk -F "," '$2 >= 40'  > filter.csv

If our values.csv file is as small as above, our filter.csv looks like this:


In fact, we could even do math with awk. Like, adding the second and the third value, divide them, those things. Check out this post about awk arithmetics.

On a side note, I’m managing to organize my posts. Almost done, when I’m done, I will resume writing my fictions here. I have now a lot to write about 🙂


Syncthing install on CentOS 7.5

screenshot-720I will start by quoting the product. What is syncthing? “Syncthing is an application that lets you synchronize your files across multiple devices. This means the creation, modification or deletion of files on one machine will automatically be replicated to your other devices.” This says it all. Next question is: What for? This can vary: I’m going to say here because it’s  multi-platform: there are apps, a web interface, a GUI, and so on, and all of if for free. Unfortunately installing it on CentOS is not for newbies. Let’s start.

Step one: create a yum repository. There’s an entry abut syncthing on the centos forum. It means to create a special repository for syncthing. What I did is I copied an already existing repository, rename it, and edit it.

cd /etc/yum.repos.d/
cp epel.repo syncthing.repo
gedit syncthing.repo

Inside the edited repo file, we copy this. Then yum clean all, yum update. Or even better, reboot if you can. At the end

yum install syncthing
systemctl stop firewalld

And if you have the browser open, the Syncthing web UI as above will open. Now what? We go to the syncthing configuration, and we edit it so that it has my CentOS client IP, not the default one.  We may want to create a service for the process, but I’m not going to tell you how to do that.

I test that I can access to the web UI from another computer, and I can. Then I install the syncthing android app (that runs on the same network than my syncthing web server) and add the device on the web interface. It’s not very intuitive: to add the device you get a QR code or a very long set of letters and numbers.  Anyway, once I add it, I see on the web UI that syncthing wants to add one of the folders of my phone to the “Folders” section. I click “it’s OK” and the sync begins. Once you are done, you have the typical options: Pause, Rescan, Edit…

I must say the final sensation is very good, so I approve it. The problem will be, as usual, to propagate and promote its usage. We’ll see how it goes!

Backdating & editing

You never finish editing. And you never finish learning and taking notes. Point taken, I find quite uncomfortable that you want to follow one story on a blog like mine, and you can’t because they are not easily tagged or arranged. So I have taken some decisions:

  1. Compile on a single ePub and edit my existing post.
  2. Edit my tags and the blog format so they are easier to follow
  3. Backdate the “computing notes” (bits) on the calendar holes.

I don’t care when my bit was published. I tend to search for the information myself anyway. I don’t know when the new situation will be stable. I do know that in the meantime, I’m also busy on my other life, that is, the non-internet one. But I’m reading you (if you have a blog) so just please wait until I’m there  🙂

CryoSPARC not starting after update to v2.8 on CentOS 7.X : bad timing interval

As usual, click here if you want to know what is cryosparc. I have created a cryosparc master-client setup. In principle I did update from v2.5 to v.2.8 successfully after running on a shell cryosparc update. It’s the standard procedure. I got updated all, master and clients. But after the update I rebooted everything. And after the reboot of the master node the problems started. This is the symptom:

cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started

And the starting hangs there. The message telling you  where to go to access to your server is not appearing. Of course I waited. The status looks like this:

cryosparcm status
CryoSPARC System master node installed at
Current cryoSPARC version: v2.8.0
cryosparcm process status:
command_core                     STARTING 
command_proxy                    STOPPED   Not started
command_vis                      STOPPED   Not started
database                         RUNNING   pid 49777, uptime XX
watchdog_dev                     STOPPED   Not started
webapp                           STOPPED   Not started
webapp_dev                       STOPPED   Not started
global config variables:
export CRYOSPARC_DB_PATH="/XXX/cryosparc_database"

It looks like in this cryosparc forum post. Unfortunately no solution is given there. We can check what the log webapp is telling also:

 cryosparcm log webapp
    at listenInCluster (net.js:1392:12)
    at doListen (net.js:1501:7)
    at _combinedTickCallback (XXX/next_tick.js:141:11)
    at process._tickDomainCallback (XXX/next_tick.js:218:9)
cryoSPARC v2
Ready to serve GridFS
      throw er; // Unhandled 'error' event
Error: listen EADDRINUSE
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at Server.setupListenHandle [as _listen2] (net.js:1351:14)
    at listenInCluster (net.js:1392:12)
    at doListen (net.js:1501:7)
    at _combinedTickCallback (XXX/next_tick.js:141:11)
    at process._tickDomainCallback (XXX/next_tick.js:218:9)

It looks like a java problem (EADDRINUSE stands for address in use). So which java process is creating the listening error?

I clean up as suggested on this cryosparc post,  or on this one, deleting the /tmp/ and trying to find and kill any supervisord rogue process. That I don’t have. Next I reboot the master but the problem persists. Messing up with the MongoDB does not help also. What now? The cryosparc update installed a new python, so I decide to force the reinstall of the dependencies. It is done like this:

cryosparcm forcedeps
  Checking dependencies... 
  Forcing dependencies to be reinstalled...
  Installing anaconda python...
..bla bla bla...
 Forcing reinstall for dependency mongodb...
  mongodb 3.4.10 installation successful.
  Completed dependency check. 

If I believe what the software tells me, everything is fine. I reboot and run cryosparcm start but my “command core” still hangs on STARTING. After several hours of investigation, I decide to take a drastic solution. Install everything again. Then I find it.

 ./ --license $LICENSE_ID \
--hostname \
--dbpath /my-cs-database/cryosparc_database \
--port 39000
ping: bad timing interval
Error: Could not ping

What is this bad timing interval? I access to my servers via SSH + VPN, so it could be that the installer can’t handle the I/O of such a load, or the time servers we use, or something. Or maybe is that the Java versions differ? In any case, I approach to it on another way. I need to be closer. How to?

I open a virtual desktop there and in it, I call an ubuntu shell where I run my installer. Et voila! bad timing gone. And the install goes on without any further issues. Note that I do a new install using the previous database (–dbpath /my-cs-database/cryosparc_database so that everything, even my users, are the same than before 🙂

Long story short: shells may look the same but behave differently. Be warned!

Perl to Python, shell to perl, python to C : about code converters

First you need to have the need to convert the code. Why to convert a piece of code from one language to another? I going to name a few reasons:

  • Familiarity. Let’s say you are just a lamer, and yiu know by heart only python, C, or FORTRAN, and you get your code on another language you are not fully fluent. You can run a converter, then check the output on the language you control.
  • Integrability. The algorithm, the function, or whatever it is, needs to come together with other pieces, written on that “other” language. Although of course it it possible to have some kind of suite written in several languages, everything is more readable and beautiful if it’s under a common grammar.
  • Portability. A lot of operative systems have shells, or something very similar or compatible. We can’t say the same of python and perl, although if you are a good programmer you could install the interpreter you need beforehand. Like if you need an specific python to run your script.
  • Speed. Speed? Yes, speed. The same compiled code for simulation running on C++ may take 10 times less running as a FORTRAN compilation. I don’t have the numbers for python versus R, but definitely, some solutions are better than others.

I say convert, not translate, since what I want is the functionality. I got a piece of perl code of unknown value that I plan to use from a bash shell. As a first step, I want to translate it. So I google about it. I found this sh2p code. It does the opposite of what I want (shell to perl) but let’s install it. To do so,

# > perl Makefile.PL 
Checking if your kit is complete...
Looks good
Writing Makefile for App::sh2p
Writing MYMETA.yml and MYMETA.json

Now we make it

# > make
cp lib/App/sh2p/ blib/lib/App/sh2p/
...some more here
cp bin/ blib/script/
/usr/bin/perl -MExtUtils::MY -e 
'MY->fixin(shift)' -- blib/script/
Manifying blib/man3/App::sh2p::Builtins.3pm
Manifying blib/man3/App::sh2p::Handlers.3pm
Manifying blib/man3/App::sh2p.3pm
Manifying blib/man3/App::sh2p::Trap.3pm

And they ask us to run a test also like this:

# > make test

PERL_DL_NONLAZY=1 /usr/bin/perl 
"-MExtUtils::Command::MM" "-e" 
"test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/App-sh2p.t .. ok 
All tests successful.
Files=1, Tests=10, 0 wallclock secs 
( 0.03 usr 0.01 sys + 0.06 cusr 0.01 csys = 0.11 CPU)
Result: PASS

Finally we install it:

make install
Installing /usr/local/share/perl5/App/sh2p.pod
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/perl5/App/sh2p/
Installing /usr/local/share/man/man3/App::sh2p::Handlers.3pm
Installing /usr/local/share/man/man3/App::sh2p::Trap.3pm
Installing /usr/local/share/man/man3/App::sh2p::Builtins.3pm
Installing /usr/local/share/man/man3/App::sh2p.3pm
Installing /usr/local/bin/
Appending installation info to /usr/lib64/perl5/perllocal.pod

MY test run (on a CentOS 7 client):

# **** INSPECT: sleep replaced by Perl built-in sleep
# Check arguments and return value

And everything seems to be correct. Nice! We have a working shell to perl translator. How about the other way around? I didn’t find anything, but there is one perl to python translator on this github repo. I clone it, download it, whatever, and I run it over the perl script I just created ( , but the results are meaningless.

Let’s check more translations. How about making an executable with pp? No, it doesn’t seem to work. But this web here seems to do the trick. Even to C,C++ and with incomplete parts. I can now cut and copy what I want into my new project! And…that’s it for today, have a nice weekend!

CentOS sudo bash no remote display

Case scenario: someone gave you sudo rights on a computer you can log in as an user. You do ssh -Y username@remote and you get a shell where you can call GUIs, for example “emacs” and you get a pop-up window (“x11”). But if you go as sudo, the shell drops you this error (or one very similar)

> nedit
nedit: the current locale is utf8 (en_US.UTF-8)
nedit: changed locale to non-utf8 (en_US)
X11 connection rejected because of wrong authentication.
NEdit: Can't open display

On this case, for nedit, that I know it’s there and I can open as “username”. What can I do to get my X11 window? Solution found, as usual, on this stackexchange post. As an user, I get the value of the DISPLAY variable, then the xauth list:

[user@remote~]$ echo $DISPLAY
[user@remote ~]$ xauth list
remote/unix:25 MIT-MAGIC-COOKIE-1 45673fghdfghd7
remote/unix:32 MIT-MAGIC-COOKIE-1 iiteiket6478787
remote/unix:29 MIT-MAGIC-COOKIE-1 b7unjyuiojnko78
remote/unix:11 MIT-MAGIC-COOKIE-1 e4t1363d166t636
...a lot here
remote/unix:14 MIT-MAGIC-COOKIE-1 e86sdaasdsyuuyu
[user@remote ~]$ sudo bash
root@remote ## > 
xauth add remote/unix:11 MIT-MAGIC-COOKIE-1 e4t1363d166t636

That is, you need to find the magic cookie for the user, and add it as sudo. This worked for me. Will it work for you? I hope so 🙂

LLDP Link Level Discovery Protocol on Windows and Linux

ldwinI need to know where are my computers connected without running around in the building, checking cables and reading not-so-easy-to-read cryptic labels. I’m not Central IT, so I don’t have access to the core switches. What can I do? First I build a list of Linux clients, like client01…client10, and Windows Clients. On then, I install the package lldpd. For CentOS 7.X, it is done like this:

yum install lldpd
systemctl enable lldpd.service 
systemctl start lldpd.service

You can now try it. For example I will show the neighbours:

root@client01 # lldpcli show neighbors
LLDP neighbors:
Interface: enXXX, via: LLDP, RID: 2, Time: YYY
ChassisID: mac AA:BB:CC:DD:EE:FF
SysDescr: My-Swit-Model-With-SW-Version
TTL: 120
MgmtIP: 111.222.333.444
Capability: Bridge, on
PortID: ifname 1
PortDescr: TP-1

Obviously I edited the output. We can grep the output, for example, on a for loop to get the SysName or PortDescr, once we are familiar with the ouput. What about Windows? For Windows, we can install the program above. You can find it here. By the way, the screen capture is from the blog entry. Now time to assemble the output on a web. Because I love web pannels 😀