Slurm controller configuration

Slurm seems to be a hot topic, and a lot of people arrive here to check it out. So I feel like by demand I need to post about a little bit more.  To the beginners I’m going to save some search and send them to my first post Slurm on CentOS 7. Now that you know how to install it, and you (may) even have a basic partition system working, let’s discuss some aspects the slurm.conf file. For now one, please remember to format the file properly, since I just show you my notes, not working solutions.

The hard core of  slurm is the resource management. Our resources are the nodes. Let’s say we want to add 3 nodes, with different features, but consecutive names. The obvious approach is to list them. Like this.

# COMPUTE NODES
NodeName=node101 NodeAddr=192.168.1.1 CPUs=20 State=UNKNOWN
NodeName=node102 NodeAddr=192.168.1.2 CPUs=40 State=UNKNOWN
NodeName=node103 NodeAddr=192.168.1.3 CPUs=30 State=UNKNOWN
# PARTITION NAMES
PartitionName=test Nodes=node101,node102,node103 
Default=YES MaxTime=INFINITE State=UP

We can also declare them in a compact way.

# COMPUTE NODES
NodeName=node[101-103] NodeAddr=192.168.1.[1-3] State=UNKNOWN
# PARTITION NAMES
PartitionName=debug Nodes=node1[01-03] 
MaxTime=INFINITE State=UP

Both ways leave slurm to manage more or less freely the node resources. This ends up on a node-locking configuration. If you submit a job, in principle, it will go to one node, and run there. The node will be then marked as allocated, and no more jobs will be allowed to go to the machine. Event if there are free resources available! So what to do instead? I suppose you have some knowledge of your hardware, so the answer is to declare the resources per node. You can still use the compact notation. Let’s say we want to add now two more nodes nodea,nodeb with the same hardware, that includes two GPUs per server. Our slurm.conf configuration should look very similar to the next.

# SCHEDULING
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
DefMemPerCPU=8192
GresTypes=gpu

# COMPUTE NODES
NodeName=nodea,nodeb RealMemory=515000 Sockets=4 
CoresPerSocket=8 ThreadsPerCore=1 
Feature=gpu,opteron Weight=10 State=UNKNOWN Gres=gpu:TXP:2

# PARTITION NAMES
PartitionName=ALL Nodes=nodea,nodeb,node1[01-03]
Default=YES MaxTime=INFINITE State=UP

What is this gres thing? In order to declare and share GPU resources, we need to add a file to the slurm configuration on /etc/slurm/. This file is the gres.conf. It looks like this:

Name=gpu Type=TXP File=/dev/nvidia0
Name=gpu Type=TXP File=/dev/nvidia1

As you see, what you declare on the slurm.conf node definition is the “type” you have on the gres.conf. In our example, TXP=Titan XP. You can name it the way you like. After being sure the new slurm.conf is on our 5 nodes, and the gres.conf on our two nodes with GPUs, we restart all the daemons and the slurm controller.

Now we have a clear resource sharing, since slurm doesn’t have to figure out how to give you the resources. The problem is, if you launch a job, you need to specifically ask for the resources you want. For example, like this:

srun -n 40  –gres=gpu:TXP:2 –constraint=gpu –pty bash -login

The above command will give you a bash login on nodea and nodeb (the ones with Feature=gpu, that we put as a constraint) where you will have 40 cores and 2 GPUs. If another job is launched in this way, it should go to the pooled resources that are free at this moment, that is, the free CPUs from the 64 in total (4x8x2=64, therefore only 24) and the free GPUs. Note that, as pointed out on my Slurm user cheatsheet, you can also “cheat” the DefMemPerCPU on slurm.conf. On the above exaple, set up to 8 GB of RAM.

What else do I need to say about slurm now? I’ll tell you: how about making a safer install, where two servers run the controller? Yes, that will be a good one. And how about the job accounting features? It will be nice to know how to manage that also. So see you soon!

Advertisements

The body function

After his awkward statement, Mu releases me. Again, without discussing it, we restart our walk, first slowly, then at a much faster path, sometimes he following me, sometimes I am following him. In 15 minutes, if my internal clock is still right, we reach the strawberry field. The last jump makes me land in the middle of it. The berries are enormous, gigantic, some of them of the size of my very head.

“Are they mutants?” I ask aloud, without waiting for an answer. Mu plucks one of the big ones and puts it inside his even bigger mouth. His munching sounds funny. “What is a mutant? Are you a mutant?”. I bite my strawberry. They are extremely juicy and soft. The feeling is like to bite a cloud, or maybe a very spongy bread. No, they are not mutants, they simply grow like this in this environment. “No, I’m a normal human.” I say.

“But normal humans are not black, aren’t they?” Surprise surprise. I’ve never expected to explain races to aliens, or whatever Mu is. “Look Mu. I explain you my color and you explain me this thing of the body function, OK?” He nods to me, very fast, very clear, repeating the movement several times, like a toy. I wait until he stop nodding, make a break, take a deep breath, and start. ” Humans are varied. We come in different colors. The reason for it has been discussed widely since the beginning of our science, and there was no agreement. Some people argue it’s an adaptative solution. Some people claim the features are reminders of our mixing with other races. I don’t know what it the truth. And actually I don’t care. But it is a fact that there is a distribution of different external features depending on your origin. Inside, we are all the same, two arms, two legs, one heart, etc. “Mu smiles. Maybe he doesn’t have a heart? ” Probably the places you visit before meeting me were inhabited by other type of humans, that’s why you didn’t meet a black person before. Where have you been previously?”

“I don’t know, here and there… but I never met somebody with your skin color.” I don’t know how to take that sentence, neither. As far as I know, the population of Africa fell in the same way than the others. Then you have also the Afro Americans. The ones that survived the nuclear bombing. Or maybe I missed some event, or simply, and most probably, Mu knows only asian areas. “So can you explain me now what is that thing of the body function you were mentioning?”

“Yes, of course, my friend.” I feel uneasy when he refers to me as a friend. I don’t know if he’s joking, he feels it, or he doesn’t know a better way to address me in a conversation. “It’s very simple. You are mass. ” I nod. “Mass is everywhere, or everywhere I know. ” I nod again, and bite my giant strawberry. It’s a little bit cooked by the sun, so the liquids the berry releases are slightly alcoholic. Another nice surprise, I’m getting drunk. He continues. “So are you. The difference between this strawberry here ” – he throws it up in the sky to then, opening his mouth, eating it without even a gulp – “and you is that you know where you are. ”  Now he stands up, and starts walking slowly in a circle. “So if I know where I am, I know where I want to go, and I know it is possible for me to be there, why can’t I be there?” He is now by my side, grinning. I grin back, the most ironic way I can. I think I got it. I try to express what he said with scientific words. “That is, in other words, your body function is allowing you, with a certain probability, to exist in a place different from this one if there is a probability, different from zero, for you, to be there.” Mu looks at me. His grin vanishes into something that I could only call indifference. “That’s what she said!” He says. We laugh.

Mu

“How can we get one? And where are them?” I remember something about the lack of oxygen being hallucinogen. This is the right place to take into account those effects. I landed here, and I stayed low for a while, therefore I didn’t feel it up to now, when I approached higher lands.

“Do not worry, my dear friend.” Mu speaks English in a funny way, like using phrases out of old advertisements, with a little accent that reminds me of India. Obviously English is not his mother tongue. “If our destiny is to find a transporter, we will. Sometimes the pleasure is on the trip, not on the destination.” I repeat my questions again. Several times. He doesn’t seem to care. We walk more. I jump, and a second later, he does it. In the middle of the jump, he flickers, like when I met him, and comes back to focus again in a couple of seconds, just before reaching the ground. “They are not far from here, less than 50 klicks, on the border of the Mare Serenitatis. “50 kilometers is a lot. Nothing that I can not cover, but not a walk. He seems to realize my concern. “Jesse, are you tired, do you want to make a break or something?”

“Wait a minute. What is going on?” Maybe because of my current mind frame, I processed the information about the 50 clicks but I ignored the fact that he simply popped out of existence and popped back with new information. “Can you move around the space?” I don’t find a better way to express what I think he did.

“You mean do this? ” Mu disappears from my side, and a waving monster with horns materializes around a hundred meters in front of me, just to vanish again before I focus my eyes. Alone for a surprisingly long minute, I call him. But when I turn my head, there he is, to my right. “Here I am!” We keep an awkward silence for a few minutes.” I see.” He laughs, open mouth, letting me see all the glory of his vegetarian teeth. ” He he he he. So you’re not aware of your body function, aren’t you?” I stare at his big eyes. He looks at me, smiling in his non-human way. We don’t stop walking, and I don’t ask for an explanation. We just walk and walk and walk. Is he going to tell me? Shall I ask?

“Shall we make a break? I feel some strawberries not so far from here. After the next hills.” I nod. Moon strawberries. So I was right. “I can get you another rabbit, if you need it. It’s not for me, you know, we are…”

“Vegetarians, you told me. But that and your name is all I know. ” He could be leading me to a trap, as far as I know. I shiver. I need to risk it. “Let’s make a break. But before we make it, I want you to tell me the truth. Why did you come to me?”

“Jesse, my friend.” I think Mu is trying to show me a serious face. Since it’s the first non-human I meet, I don’t know if this is what he really wants to express. “I came to you because I heard your call. You need a partner, here I am!” He waves his arms around his body, in a circular gesture with a clear meaning but with a little comical result. “Didn’t you feel alone here on the moon?”

“Yes I do.” Still walking, he comes closer, and slowly wrap his arm around my waist. I don’t resist. His big hand feels cold over my shirt. He grabs me tighter. We stop, and face each other. He puts his other hand in my waist, like sustaining me, gently. I let him do. Now he has me the way a man holds a lover before a kiss. I smile with the thought. He smiles back. “Then it’s settled. Welcome partner, welcome home.”

Walking on

“Giant steps are what you take
Walking on the moon
I hope my legs don’t break
Walking on the moon
We could walk forever…”

I miss my SSD Player. I didn’t have time to recover it before I got dumped by Machiko. Gosh, I guess it is the most memorable dump on the history of mankind. I’ve been dumped to the Moon! I wonder if I will make it to the Guinness record book, or something similar.

So I sing my own songs, while walking. Now that I have the stomach full, I don’t jump. I have a good memory, therefore I prepare a moon themed playlist. Walking on the moon, The Police. Man on the moon, REM. Moon over bourbon street, Sting…and in the meantime, I forget where I am. I feel a little intoxicated. Was it the moon rabbit? Was it the green goo? The water? I consider sitting…

“Sittin’ in the mornin’ sun
I’ll be sittin’ when the evenin’ come”

But I know by experience that, if you’re under the effects of an unknown substance, the best is to continue acting as if the thing was not in. This is already a trippy place, therefore the only thing I need to do is not to take it as serious as before. For example, that big grey guy with horns that seems to walk with a steady path in my direction must be obviously an effect of the intoxication. And now he walks with me. The mountains don’t move. He speaks.

“Where are you going?” He’s humanoid only. It reminds me something out of a japanese tale, some kind of oni. He’s also dressed like one of them, so I ask. “Are you an oni?” He looks at me, undisturbed, and keeping my walking speed, before answering. “Are you a gaijin?” I nod. “Then I am an oni.” I smile. Why am I so relieved? I just met a mythological being, and he spoke to me. “Are they all here?” I ask my oni. “It depends of what you call all ” it’s his surprising answer ” I saw a few elves over there, and an unicorn. But they don’t seem to be very solid. I tried to eat one the other day…and he popped out of the existence.” I smile like a stupid. Should I have laughed? Was that a joke? We keep walking. “What an uncourteous thing to do, to disappear without any goodbye.” He looks at me. Or I think it’s a he. Suddenly, he starts laughing. His mouth is very big, and he has two big fangs in addition to the pleyade of pointing teeth. “He he he he. I like you. ” I tremble, maybe I’m his moon rabbit and he likes to speak with his preys before lunch.

“Don’t worry, I’m vegetarian.” He says, like after reading my mind. “These fangs ” he grabs then with his big hands “are an anachrony, our kind don’t need such an appendix anymore. I was thinking about cutting them off, as a lot of people are doing in the City, but I’m afraid of the pain… so I keep them.”

My trembling is gone. “I’m Jesse” I say. “You can call me Mu ” is his answer.  We shake our hands, still walking. Don’t stop. In my head, Journey suddenly starts playing.

“Just a small town girl
Livin’ in a lonely world
She took the midnight train goin’ anywhere..”

“Where are you going, Jesse?” I answer babbling incoherences, about that I’m not supposed to be there, about Machiko and about my desire to come back. “Well, maybe I can’t even come back to my Earth, but I will not be sure until I try it.” Mu looks at me. “Dear Jesse, it is possible to come back. Or it was. What you can’t do is to break the local space-time continuum…the non-local, maybe you can. Do you copy me?” I nod. “But let’s fix one problem at a time. First we need to get a transporter!”

Qt5 and FOCUS install on CentOS 7

FOCUS is the interface between data collection and data processing in cryo-EM. Or that is what they say. As you know if you arrived here, there are dozens of other tools to process cryo-EM data. But let’s go through my installation log. By clicking on download I’m offered the chance to get some Linux packages, since I run on Linux. I get the RPM and I place it on my test machine. Since it’s a RPM, I try with rpm. The output is below.

 rpm -Uhv focus-1.0.0-stable-Linux\(RPM\).rpm 
error: Failed dependencies:
 libQt5Concurrent.so.5()(64bit) is needed by focus-1.0.0-1.x86_64
 libQt5Core.so.5()(64bit) is needed by focus-1.0.0-1.x86_64
 libQt5Gui.so.5()(64bit) is needed by focus-1.0.0-1.x86_64
 libQt5Network.so.5()(64bit) is needed by focus-1.0.0-1.x86_64
 libQt5OpenGL.so.5()(64bit) is needed by focus-1.0.0-1.x86_64
 libQt5Script.so.5()(64bit) is needed by focus-1.0.0-1.x86_64
 libQt5Widgets.so.5()(64bit) is needed by focus-1.0.0-1.x86_64

Which is expected, if you read the documentation. We need Qt5 for the GUI. Knowing that yum can install also rpm, I try it. This is what you get:

yum install focus-1.0.0-stable-Linux\(RPM\).rpm 

Installed:
 focus.x86_64 0:1.0.0-1

Dependency Installed:
 qt5-qtbase.x86_64 0:5.6.1-10.el7 
 qt5-qtbase-common.noarch 0:5.6.1-10.el7 
 qt5-qtbase-gui.x86_64 0:5.6.1-10.el7 
 qt5-qtscript.x86_64 0:5.6.1-10.el7 
 xcb-util-renderutil.x86_64 0:0.3.9-3.el7 
 xcb-util-wm.x86_64 0:0.4.1-5.el7

Complete!

Unfortunately, I’m not able to get the executable to run. So I go the hard way, undoing the yum install and getting the source code. I’m going to follow what is available on the FOCUS wiki. With my lovely comments added, when needed.

yum install git
git clone https://github.com/C-CINA/focus.git
Cloning into 'focus'...

I move it to our /usr/local/ and try directly to build it:

/usr/local/focus ## > ./build_all
...bla bla bla...
No GPU-support
NOT USING CUDA
-- The variable FOCUS_ROOT_DIR is set to: 
/usr/local/focus
-- The variable FOCUS_INCLUDE_DIR is set to: 
/usr/local/focus/include
...bla bla bla...
CMake Error at cmake/FindQt5.cmake:75 (MESSAGE):
 Qt5 NOT FOUND:
qmake installation NOT FOUND!

-- Configuring incomplete, errors occurred!
See also "/usr/local/focus/build/CMakeFiles/CMakeOutput.log".
Configure failed for Focus

I try to fix the problems, starting with the Qt5. I check which qt we have and I try to update it with yum. See it below:

## > yum install qt
 ...
 Package 1:qt-4.8.5-13.el7.x86_64 
already installed and latest version
 Nothing to do
 ## > yum update qt
 ...
 No packages marked for update

The installation of qt-creator (yum install qt-creator) also doesn’t result on a successful build. So I download Qt, and try to install it from scratch. The package I download is called qt-opensource-linux-x64-5.8.0.run. When you make it executable and run it, a nice GUI pops up to ask you  at the very beginning for a Qt account details. I skip that, and change the installation destination so it goes to  /opt/local/Qt5.8.0, but the rest I leave. An “ls” over my folder reveals indeed I have the desired “qmake” on /opt/local/Qt5.8.0/5.8/gcc_64/bin/. My qt5.sh file looks like:

## /etc/profile.d/qt5.sh
PATH=/opt/local/Qt5.8.0/5.8/gcc_64/bin/:$PATH
export PATH

I run it, as suggested.

 ./etc/profile.d/qt5.sh
## > which qmake
/opt/local/Qt5.8.0/5.8/gcc_64/bin/qmake

Now I come back to my installation, and run it again.  Unfortunately, the script fails at 90% with this error:

[ 90%] Building CXX object 
apps/fLogBrowser/CMakeFiles/fLogBrowser.dir/LBMainWindow.cpp.o
In file included from /usr/include/c++/4.8.2/type_traits:35:0,
 ...bla bla bla..
 from /usr/local/focus/apps/fLogBrowser/LBMainWindow.cpp:20:
/usr/include/c++/4.8.2/bits/c++0x_warning.h:32:2: 

error: #error This file requires compiler and library support 
for the ISO C++ 2011 standard. 
This support is currently experimental, 
and must be enabled with the -std=c++11 or 
-std=gnu++11 compiler options.

Therefore I need to go for an older qt5. I get it from here. Then I change the corresponding scripts, and launch the install again. Unfortunately it fails again, with a quite dark Qt compilation error that doesn’t seem to be easy to solve, unless I have full control of the code. The problem is, Qt5 seems to have been heavily modified from version to version in recent times, to be adapted to run on Android, and things like that.

I have no other option than to try again the RPM package, now with my local Qt5.0.2.

yum install focus-1.0.0-stable-Linux\(RPM\).rpm

After the “Complete” message, I look for a working executable, that I find on /opt/focus/bin/focus.  It seems to work now, time to test, although I’m afraid this is not the end of it. We’ll see what’s happening with the CUDA support, for example 😛

EDIT: I found out a very efficient way to install it, on the place I want.

yum install qt5-qtbase qt5-qtbase-common \
qt5-qtbase-gui qt5-qtscript xcb-util-renderutil xcb-util-wm

rpm -Uhv --prefix=/usr/local/focus/ 
focus-1.0.0-stable-Linux\(RPM\).rpm

So first you install qt5 via yum, then focus via rpm 🙂

Not yet there

I don’t have clear what’s going to happen with Jesse and Machiko, and also the plot of the story. I don’t even have a title for the project. But I have an universe, a big scene, so maybe I can simply let my people evolve there, add some knots, and see what’s happening to them. To write at least 200 words per day is not a big challenge, the challenge is to write them in a coherent way, so that the reader is fatally engaged by the first ten words and surprised of the end when the entry is done. So that if I pack the posts all together, the result is a good, coherent, long read.

Actually, this is my third long plot, my third universe. The first one is the one of The echelon. The second starts with 1992. Both are unfinished, and they will stay like that unless I have a new dream to complete them, or I have time to pack them in an eBook form. But knowing myself, I will probably start yet another three plots, before coming back to the previous, blocked ones. What can I say? I’m an amateur, I don’t get money from it…and English is not my main language 😛

Food

I’m craving for strawberries. The ones from my childhood, the ones I was stealing from the greenhouse plantations. It’s been years since I try some. In the System, the taste of food is not a priority. It’s not like in jail, but you get the calories you need, nothing more, nothing less. If you don’t need a cake, or sweet strawberries, you don’t get them. And who really needs a cake? Politicians, diplomats, maybe. In a similar way, you don’t eat what is not supposed to be eaten by you. Like oysters. If you steal food, your right to choose will be removed. Yes, of course you do have food choices. Good food is happiness. It’s Opium für Volk.

Take for example the Spanish omelette. Or the Paella. Of course, they are no more Spanish dishes, since Spain is no more. The land is there, but not the borders or the identity. The country recipes are anyway good, and they are not complicated. So they give us that. The system has a giant Mensa: you will have either spanish omelette or schnitzel, but not spaghetti, if they don’t offer it. I have heard of people that cook, after weeks of collecting ingredients, and I also heard of clandestine restaurants. What a crazyness, to risk your life for a new menu.

Thinking about food is making me hungry, more and more.

So where are the moon animals? Am I the only one? I’m so hungry that I could eat a transporter, and knowing that they are some kind of giant vegetal whales is not making the situation easier. And where is the string I can see from the Yellow Earth, the string connecting both planets? Why can’t I see it now? If I find the string, I can find a transporter, and over it, maybe humans, humanoids, or simply a good conversation.

I jump as much as I can. I go up to 10 meters over the surface, I guess. Distances are tricky on a thin atmosphere, your eyes can deceive your brain. No, not a single move on the surroundings. I fall 50 meters away from the water crater, and before start walking again, I perform a test digout on the green goo.  Maybe I can find worms, or some similar insect. Worms are delicious on Earth, if you know how to fry them. I scratch, and scratch with my bare hands… but no luck. Not a single moving thing that I could appreciate with my bare eyes. No insects at all. The thin compost carpet covering this part of the Moon may be created only by local bacterias.

Before I escaped, my previous mission was to check against mine the academic level of the basic system education. Now everything is done with computers. They teach you to trust them.  I don’t need to pass through this chain of thoughts, since I was born analoge. That’s another advantage for me here versus a younger system minion, if they ever care to chase me.  I don’t trust on computers, I trust on math. I could, if needed, even solve a wave function, given the right parameters. I know the magic. I don’t need a computer to find the eigenvalues, just time. I was able to solve with a pen and a paper such a calculation, like diagonalize a matrix, when I was 10.  The problem I’m facing now could also be solved with math. The numbers behind life are quite simple in principle: water + oxygen + energy.  The math for complex life requires more factors, like the need to evolve, or the need to compete for a certain set of resources. If I want to find something more complex than the green goo, I need to explore the factors, I need to go to a more hostile piece of land. I need to reach the mountains.

Before I reach them, when I’m midway there, around 7 kms from my landing point, I saw it. From the privilege of the height of a jump, the first moon creature run away right under my feet. On a quick move, I managed to capture it, and even more complicated, I managed to start a small fire to cook it. I simply covered it with green goo, and threw it to the hesitant flames. I just finished devouring it. It turns out that the moon has indeed rabbits, and that they taste as delicious as their terrestrial cousins.