Happy three friends

“Let me introduce you to the people: the girls are Lana, Yomick, Bru and Alea, the boys are Pit, Angel, Ber, and Zap. The other one is Ea. Ea is not defined, you can call him or her whatever way you like. We simply call Ea, Ea.”

Fantastic crew! I like them all! I start by kissing the girls, and hugging the boys, then I stop in front of Ea. Ea seems to realize my confusion and he/she first hug me, then kiss me. In the middle of us, over a square, little indented, little twisted piece of masonry of the size of a small table, there’s an open barrel of around 50 liters of something. Hanging around it, a variety of drinking containers lay also: a horn, a glass, a teacup, a mug, a wooden vessel. I’m offered something that looks like out of a souvenir shop, since it’s quite baroque and it depicts the message “Frankfurt” written in gothic letters. Without doubting a second I sink it in the barrel, and I bring it full to my lips. My throat burns. Is it the cone I just ate?

“I propose a toast!” Red takes the mug, and does the same gesture I just did. The others repeat the ceremony, one after the other, in a similar way.  Red speaks when everybody is ready. “For LeBleu, the best challenge rider on Yellow Earth!”

“For LeBleu!” They shout. I quickly look around in the search of public, but the level of noise we’re generating doesn’t go so much above the background. We’re just a tear on a sea of sounds. Voices. Laughs. Other shouts. Clings, clashes. Over all I hear the drums coming from everywhere, their beats mixing up in the way only the drums can do, like in a conversation: one group beats, another answers. Drums in the dark. We’re in the middle of a relatively disperse crowd, but I seem to hear also people having sex close to us. Am I imaging it? I finally gulp the liquid in my glass, it’s first sweet, then sour, then something else. Tasty. Just what I needed after our long walk. Again I remember Adam. Where is he now? I try to speak.

“Tell me, Yellow, where do you watch the movies?” If they watch movies here, maybe they record them also, and I have always an urgent need of attention. The blonde girl looks at me, then to the dark haired Zap, that is now grabbing her waist. “There’s a viewing inside.” Zap says.

“Inside?” I point to the Tower, that shines now in a multitude of points, like in flames. The flames get more scarce the higher you look. Some of the faraway sparks seem to move up slowly, like crawling. Actually from here the Tower waves high above our heads, like a bamboo tree under heavy wind, but there’s no wind. A dimensional effect?

“Inside. Are you going to leave us so early?” Ea speaks. As an androgyne, his/her voice sounds very deep, so probably Ea was born a man. I can’t avoid but look to his (her?) crotch, and Ea seems to realize it. “If you leave, you’ll never know what I have here!” Ea says. It sounds like a challenge. But I don’t know if I will take this one. Besides, is it really a challenge, or is it what I want?


The twins

“Wow, she’s here!” Two teens come running to me, when we start to get close to the mass of party people. “Can you believe it? LeBleu is here!”

“Who are you girls? Do I know you?” My guide up to now steps backwards. Adam seems to have problems to cope with the obvious intoxication of the twins, that grab me, touch me, and jump around, like if they are on the presence of  a prince, a millionaire, or something like that. Like a fan.

“We watch you when you did the jump on kernel Munich! That was amaaaazing! Did you know they recorded it in celluloid?” Now that explains the man over the wooden tower, pointing me with something like a very small canyon with two wheels on the top. I was concerned but I decided to let it be and jump. The thing was a portable mechanical film camera! So it is true: they’re starting to record old style movies here. I discarded the idea when I saw the film camera because of the well known irregularities and the way the technology fails. Damn, even the electricity is not working over a long distance. But I guess if you’re on the right zone, maybe the photons will manage somehow to imprint the celluloid in an uniform way… and if you have a mechanical generator as the ones described on the books… it is definitely possible. That opens a brand new business for me. Maybe I can become a yellow earth youtuber, after all. I need to watch that tape!

“No I didn’t know I was being recorded! Otherwise I charge them! So tell me, what are your names? And what can one challenge rider do here?”

“I’m Red, and she’s Yellow.” They make a pose in front of me. “We named us after your nick!” they shout. “You are the best of the tapes. Come!” I’m grabbed by both arms, Red to my left, Yellow to my right. We should make a colourful picture for Adam. By the way, where’s him?

“You need to try this mushroom! ” Yellow withdraws some kind of pouch from under her pants. From inside, she takes one round cone, of the size of my thumb, of a yellowish color. Probably dried. Tentatively, I grab it, and smell it. Without asking about, I eat it. What else could I do with it? “Amaaazing! ” Again they speak at the same time. “LeBleu is taking cones with us! What else can happen tonight?”

Slurm vs torque vs SGE

It looks like slurm is becoming more and more popular. I must say I like it also, but I need to comment on other queuing systems that were there before.
For example, torque ,SGE (Sun Grid Engine) or LL (IBM Load Leveler).
From the user’s point of view, here you have a comparative table that I found around:
TORQUE command SGE command LL command
Job submission qsub [scriptfile] qsub [scriptfile] llsubmit [scriptfile]
Job submission to a node qsub -l nodes=sbnode102 [scriptfile] qsub -l nodes=sbnode102 [scriptfile] # @ node = nr.of.nodes
Job deletion qdel [job_id] qdel [job_id] llcancel job_id
Job status (for user) qstat -u [username] qstat [-j job_id] llq -u [username]
Extended job status qstat -f [job_id] qstat -f [-j job_id] llq -s job_id
Hold a job temporarily qhold [job_id] qhold [job_id] --
Release job hold qrls [job_id] qrls [job_id] --
List of usable queues/classes qstat -Q qconf -sql llclass

Source: here

As you see, the user experience is very similar. I didn’t try LL simply because you need to pay it. I think SGE is no more, and you need to use now something called Univa.
My first choice for a queuing system was torque. I chose torque somehow on a egoistic base: I used it before, therefore I was expecting a shorter learning curve.
The torque installation mechanism is similar to slurm  (you make your rpms, then you copy them on all the nodes) unfortunately, due to its multiple dependencies (like openmpi) it is a little bit more cumbersome and not easy to maintain.

To list the nodes, on the login node type

 pbsnodes -a 

A sample output will show a long list of nodes with information about them. A more readable output can be shown by typing:

 pbsnodes | grep -v status | grep -v ntype | grep -v mom 

The last 50 entries on the cluster server log of a specific day are visible by typing:

 tail -50 /var/spool/torque/server_logs/YYYYMMDD 

The node logs for an specific day can be checked in a similar way:

 tail -50 /var/spool/torque/mom_logs/YYYYMMDD

To show server parameters, like walltime (also as user) type:

 qmgr -c 'p s'

qstat command gives a list of jobs which looks something like this:

Job id            Name      User   Time      Use S Queue
----------------  ---------- -----  -------- - -----
1121.headnode1    jobName1   bob    15:45:05 R  priorityq       
1152.headnode1    jobName2   mary   12:40:56 R  workq       
1226.headnode1    jobName3   steve  0        Q  workq
So far so good. So why slurm and not torque?
Firs you have the job management. The jobs are better handled, they can run independently of the openmpi installation, and if you compile slurm with the right options, you will be able to store the job results on a MySQL database.
Second you have the software dependencies. If you need to use a specific openmpi (and I need to) you need to have torque compiled with it. That is not the case of slurm, that can work with the corresponding module.
So which one is your favorite? Or even better, which one do you think is performing better? I don’t have numbers…but I think slurm is here to stay 🙂

The Chaos

I’m born here on Yellow Earth. So I heard about things like the internet, mosh pits, mega cities, terrorism and the nuclear holocaust like if they were fairy tales. I know due to the time asynchrony some people just arrived here, to a strange new World with a green moon and no apparent government or currency. I must add, I’ve never considered myself also a believer of the Master Plan, that will make us Humans Master of the Universe.

I think about all those legends of the original Earth, the mosh pits, the megacities, when I see what is going on around the Tower.

The Tower stands on the geometrical center of a square. The square is therefore interrupted by its mass, and everything that hangs from it. It looks more like a gigantic tree, like if it was grown there instead of built. And maybe it was. Two gothic arcs at different heights are predominant on both sides of it,  although not symmetric, they seem to be built on the same style. There are temporary structures hanging of them, filled with candles or gaslights, defying gravity, reminding me of spiderwebs illuminated by fireflies. Under them, the Chaos is unraveling. Under the left spiderweb umbrella, the lower one, loud music plays, drums, guitars, and voices can be heard, mixing in a casual way. The beat coming, even muzzled by the distance, is contagious, and I need to stop myself to run away there. I can distinguish shining naked bodies around big things (painted animals?) jumping in the middle. What are those, oversized kangaroos? To the right, there are towers of barrels, spraying alcohol over the crowd. Some people seem to be taking care the spray is reaching everybody. I know it’s alcohol because of the smell! The smell is so strong that is reaching where we are, still a hundred meters away from them. The multitude seems to be ready to die out of an overdose of whatever they are providing, and probably they are. Oddly enough, from my point of view, everything seems to be manageable: it’s a big party, right, but nothing you can’t cope with, especially if you’re a challenge rider.

Install munin-node on CentOS 7

As you know, I was installing a new system (the technical word I think is provisioning) on an old server (see Gtk-WARNING ** cannot open display on CentOS 7 minimal server install). As a part of the installation we tend to make the new system a munin client. If you want to start from scratch, with a munin server installation, you can follow this howtoforge tutorial. Before, we more or less installed a A Windows munin node, so it’s time to comment on the CentOS version. No rocket science here also 😀

I will suppose you’re with the latest kernel already and no graphic desktop, although if you are behind or beyond this point, there should be no problem with this procedure also. First we install munin via yum.

yum install munin...bla bla blah...
Transaction Summary
Install 1 Package (+125 Dependent packages)

Don’t be scared about the amount of packages. Munin is supposed to monitor a lot of parameters, so it needs a lot of packages. Also, by typing yum install munin, you get also the munin server packages, in case you want to upgrade the install later. Before starting munin, you should update your munin node configuration on /etc/munin/munin-node.conf. The parameters I need are:

host_name MYSERVER
allow ^XXX\.YYY\.ZZ\.AA$ 
allow ^127\.0\.0\.1$
allow ^::1$

I assume here MYSERVER is your server’s name, known by your network, and that XXX.YY.ZZ.AA is your munin server IP address. If we start the munin service now, we will get the all kind of errors. You can check them on the log.

more /var/log/munin-node/munin-node.log 

Resolved [*]:4949 to [::]:4949, IPv6
Not including resolved host [] IPv4 because it 
will be handled by [::] IPv6
Binding to TCP port 4949 on host :: with IPv6
Setting gid to "0 0"
59416" Local: "[::ffff:SOME-IP]:4949"
[13571] Error output from fw_conntrack:
[13571] cat: /proc/net/ip_conntrack: No such file or directory
[13571] Error output from fw_forwarded_local:
[13571] Can't find conntrack information

Also, the system status should give you a similar output. This is due to two issues. The firewall must be off, or it must allow the connection with the server, and some munin plugins need to be removed or tuned up. In my case, this is done in a couple of command strikes.

> systemctl stop firewalld.service
> rsync -av other-munin-client:/etc/munin/plugins/ 
/etc/munin/plugins/ --delete-after --progress

The less plugins you have, the better the server will handle the client. This is my list of deleted plugins.

vmstat users threads swap processes
proc_pri postfix_mailvolume postfix_mailqueue
open_inodes open_files netstat load irqstats interrupts 
if_err_something if_something 
fw_packets fw_forwarded_local fw_conntrack 
forks entropy diskstats df_inode df

Time to start it, and enable it so it comes back after a reboot

systemctl start munin-node.service
systemctl enable munin-node.service

A good munin node log in my case is looking like this.

CONNECT TCP Peer: "[munin-server-ip]:57046" 
Local: "[local-ip]:4949"

Don’t be inpatient and wait at least 15 minutes so you have 3 points to show on the graphs. See you later or tomorrow!



We walk through the city for another 20 minutes, in silence, always heading more or less north, nobody to be seen. From time to time I look up, to the moon, trying to find out if we’re getting closer, but it doesn’t look like. We cross a couple of bridges, but I can’t tell they are over a road or over a branch of the river. From time to time I hear muffled voices, whispers, outside our circle of light. Animal noises. Or what I think is a door closing, or a shout. When I try to stop, he pushes me.

“By the way, how can I call you, love?” He obviously knows my nickname, but I’m not forced to ask who’s challenging me.

“You can call me Adam.” We go ahead again. The neighborhood we’re crossing looks like abandoned. No, not at all, there we have a light on a window. It’s hard to say something else in this darkness. “Is LeBleu your real name?”

“My real name? What is that? Who cares about that nowadays?”

“I care.” Is his surprising answer. “I never liked these youtube nicknames you guys are having. They’re catchy, I agree, but I can’t seriously think about having a friend called DragonRider69 or KillerBlade.” He has the point, and it’s not the first time I hear the argument.” So will you tell me your name?”

“Maria.” He smiles. We stop. A gaslight illuminates a small square, on the buildings surrounding the square, I see wooden signs hanging, but I don’t have the mood to read them. Probably shops, maybe offices. “But please, call me LeBleu, if you call me Maria in public, probably I will not answer you. Instead, I may even try to kill you in a discreet manner! ”

“This is the registration office.” He ignores my last remarks, and points out the biggest building on the square. “It’s the closest thing to a town hall we have around. I thought you could be interested on knowing where it is, in case you need to sort out something.”

“You see, if I announce on your message board that Maria will be accepting challenges the next night, how many people do you think will come to challenge me? Or what kind of people?” He laughs, but in silence, gesture that I really appreciate. Anyway, it was my choice to give Adam my real name. I could have gone for something more exotic, like Melkera, or Antigona, or Bulene. Now it’s too late, the deal is done, the door is closed, whatever.

“Alright, Maria.” I emit a loud snort to express my unhappiness. “Just testing! The last 20 minutes of walk and we’ll be there! Do you want to tell me what’s your last will before enter into the Chaos?”


“Why do you want to go to the Tower?” Suddenly he recovers his courage. “It’s an unstable area, you know it, right? You can turn to the left while strolling around and suddenly end up in another point of the City…or in another world. Who knows?”

“Are you scared of getting lost?” I squeeze his butt. He smiles, but his hands still don’t come to my body. Which is a good point for him. I think I have him, I think he will do whatever I ask if I ask him in the right way. “I just want to have a clear view of the base of it, maybe check what kind of people you find around…that’s it.”

“Do you see that little bit of moon that seems to be missing on the left? Yes, right above the string, the channel, or whatever you call it in your country. That’s the tip of it. You can’t miss it. Actually, I wonder how did you miss it in the day. Did you just arrive?”

“You got me!” I winkle the best way I can. Of course I saw it in the daylight. Actually, you can see it from far away, a curved and shaggy skyscraper marking the center of the region. “Who goes out in the day, anyway? I prefer the nights, where I do my business.” He seems to balance something on his mind. To help him to tip over on the right direction, I kiss him on the lips, briefly, like giving him a taste of what is to come.

“OK, let’s go. “He takes a torch from the many left on the bucket for that purpose, leave a coin on the box for that purpose, and ignite it. I have a last look to der dunkle Ritter. Inside, the party goes on. I wonder who’s next in the program. We start walking, on a steady speed, direction to the Tower. The noise from the bar quickly vanishes. The streets are very dark, and the clouds don’t let the starlight to pass. I look at him, waiting for more comments. He seems to realize he appears to be scared, and slow down the steps until we look like a couple in love, walking around. Suddenly he start to speak again, with a pitch that tries to be cool. “I think actually it is indeed a good plan to reach the base, it’s this time of the night they open the barrels of wine for the pilgrims. But please… in the way… do not run away, you can really end up landing on a fast path!”