A leginon CentOS 6 container on CentOS 7

Continuing with our cooking recipes, I’m going to install leginon on a CentOS 6 container running on CentOS 7. We start creating a CentOS 6 container:

lxc-create -n centos6 -t download -- -d centos -r 6 -a i386
lxc-start -n centos6

To log in via ssh we need first to install the openssh server. Also, let’s install more things that we will need if we want to have a nice leginon installation experience.

yum install -y git which openssh-server wget \
unzip tar ffmpeg ffmpeg-devel

Now we better restart the container and log in as root via ssh. We’re ready to start installing. Since we have CentOS 6 we can directly use the autoinstaller.  I’m not going to discuss the previous steps of the complete installation but go to step 6 directly. So we do

python centosAutoInstallation.py

This is retrieving and installing every package we need using different tools, so we need to monitor everything goes smoothly. In my case, the first time it didn’t finish successfully due to several packages missing. I will blame my basic container for that. I choose no sample download, no software install, and I get:

ERROR: Failed to run Myamiweb setup script.
You may try running http://localhost-something 
in your web browser.

Let’s try to go ahead and ignore the error. Next issue occurs when I try to run firefox from inside the container.

process 16203: D-Bus library appears to be incorrectly set up; 
failed to read machine uuid: 
Failed to open "/var/lib/dbus/machine-id": 
No such file or directory
See the manual page for dbus-uuidgen to correct this issue.
 D-Bus not built with -rdynamic so unable to print a backtrace
Redirecting call to abort() to mozalloc_abort
Segmentation fault

The solution to the D-Bus library problem is easy.

> dbus-uuidgen > /var/lib/dbus/machine-id

I open firefox and I check that I can see the leginon start page. This is not the end, now we need to do the database server installation.  I cut and copy from the HOWTO:

# yum install mysql mysql-server # already done by autoinstaller  
ls /usr/share/mysql/my* --> they are indeed there
cp -v /usr/share/mysql/my-huge.cnf /etc/my.cnf

I modify my.cnf as suggested, I add  default_storage_engine=MyISAM at the end of the file, and I service mysqld start, chkconfig mysqld on. So far so good.

Since we want to have a container as portable thing, instead of creating the database, we are going to import it. We do as I wrote on phpMyAdmin upload and fix old database plus an additional step:

chown -R mysql /var/lib/mysql/

Otherwise starting mysqld fails with the error:

[ERROR] Can't open and lock privilege tables: 
Table 'mysql.servers' doesn't exist

We can open phpMyAdmin on firefox now, or do the checks suggested on step 13 and 14. All looks clear. What now? We do have already a leginon server up and running on ‘real’ hardware , so instead of  runnig the online setup wizard by visiting http://yourhost/myamiweb/setup or http://localhost/myamiweb/setup to create the myami website’s config file, as described here, we simply rsync the folder from the running one to our container, and perform service httpd restart. Provided the data folder is mounted in the container, and so on, we now have a working leginon container. In principle we can stop here, but we want to update it regularly so that we have the latest software version. The first update I do manually. First we clone the beta version of myami like this:

git clone -b myami-beta http://emg.nysbc.org/git/myami myami-beta
cd /path/myami-beta
./pysetup.sh install

And we check that everything works:

~/myami-beta/leginon ## > python syscheck.py
 Looking for previously installed Leginon...
 Leginon found here: /usr/lib/python2.6/site-packages/leginon
 *** It is best to uninstall your previous 
Leginon before installing the new one. 
The best way to uninstall is to move it to a backup
location, just in case you need to revert to the old version.
 Python executable (if wrong, check PATH in your environment):
 Python module search path (if wrong, check PYTHONPATH):
 Python version: 2.6.6
 OK (at least 2.3.4 required)
 Python says home directory is: /root
 Python Imaging Library (PIL):
 importing Image module...
 PIL version: 1.1.6
 OK (at least 1.1.4 required)
 MySQL Python client (MySQLdb):
 importing MySQLdb module...
 Python MySQL version: 1.2.3c1
 OK (at least 1.2 required)
 importing numpy module...
 numpy version: 1.4.1
 OK (at least 1.0 required)
 importing scipy.optimize module...
 testing for leastsq function...
 importing wx module...
 wxPython version:
 OK (at least required)
 Testing a wxPython application. 
Close the window that pops up...
 wxPython test successful

So it seems to work. We could copy the stuff from myami-beta to the location given on the test,  /usr/lib/python2.6/site-packages/leginon. Time for the power user to test it! We’ll see how long until we screw it up 🙂



Posted in bits, centos, containers, hardware, linux | Leave a comment

CentOS 7 error while loading shared libraries: libfftw3.so.3

Case description: on my centos7 container, I sync ProgramX from the container host (centos 7 also) and when I try to run it, I get the error.

ProgramX: error while loading shared libraries: 
libfftw3.so.3: cannot open shared object file: 
No such file or directory

To fix it it’s very easy.

yum install fftw3

That’s it. Just to add a little background information, fftw are the Fast Fourier Transform libraries. They may be present as a part of other installed programs, but in principle, they don’t come by default, since fast Fourier transforms are not common system operations 😀

Posted in centos, containers, linux | Leave a comment

LXC : containers on CentOS 7 (III)

This is my container cooking recipe so far. I will call the host where the containers run ‘local‘. On ‘local‘, the initial configuration is:

### install the initial packages
yum -y install lxc lxc-templates libcap-devel \
libcgroup busybox wget bridge-utils
### check the templates we have
ls /usr/share/lxc/templates/
### create the container centos7 after the template centos
lxc-create -n centos7 -t centos
### change the initial root password
chroot /var/lib/lxc/centos7/rootfs passwd 
### install the virtualization lib, start it
yum install -y libvirt
systemctl start libvirtd
### install lxc-ls
yum install -y /usr/bin/lxc-ls

We may need to install debootstrap to download the specific container system packages, if the system is not the same than the native one.For example if we try to create an ubuntu container…

 > lxc-create -n ubuntu-test -t ubuntu
'debootstrap' command is missing
lxc_container: lxccontainer.c: create_run_template: 
1092 container creation template for ubuntu-test failed
lxc_container: lxc_create.c: main: 
274 Error creating container ubuntu-test
> yum install debootstrap
> lxc-create -n ubuntu-test -t ubuntu
Checking cache download in /var/cache/lxc/precise/rootfs-amd64 ... 
Installing packages in template: ssh,vim,language-pack-en
Downloading ubuntu precise minimal ...
---> OK

There’s another way to create a container. This is how we create a CentOS 6 container with a i386 architecture:

> lxc-create -n centos6 -t download -- -d centos -r 6 -a i386

Once created, don’t forget to chroot /var/lib/lxc/centos6/rootfs passwd  to change the default root password and yum install openssh-server if you want to ssh to your centos6 container. Anyway, we will mostly work with the centos7 container. Before making more changes we do a copy of the newly created container configuration.

cp /var/lib/lxc/centos7/config /var/lib/lxc/centos7/config.bkup

To avoid systemd making the container sluggish we edit the container configuration and add the line

lxc.kmsg = 0

Since ‘local‘ has GPUs, we add also GPU support. At the end of the container configuration file, we write:

lxc.mount.entry = /dev/nvidia0 dev/nvidia0 
none bind,optional,create=file
lxc.mount.entry = /dev/nvidiactl dev/nvidiactl 
none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm dev/nvidia-uvm 
none bind,optional,create=file

In the same way we share the nvidia drivers, we can share other mount points. This is done by adding to the configuration file something like the next, to add the /home and /data folders from the computer ‘local‘.

lxc.mount.entry = /data data none bind,optional,create=dir 0 0
lxc.mount.entry = /home home none bind,optional,create=dir 0 0

We are ready to start on local the container centos7 on  the background, so that we can do ssh to it later.

lxc-start -n centos7 -d

We can log in as root to the container centos7 from ‘local‘. That is indeed what we wanted. Since centos7 doesn’t get a name from a domain name server, we will ssh to its IP, in this case We will there install via yum additional packages and sync the software from the host computer ‘local’.

user@local ~ ## > ssh -Y root@
[root@centos7 ~]# yum install net-tools which rsync
[root@centos7 ~]# yum install environment-modules
[root@centos7 ~]# rsync -av root@local:/root/ 
/root/ --delete-after --progress
[root@centos7 ~]# > rsync -av \
root@local:/usr/share/Modules/modulefiles \ 
/usr/share/Modules/modulefiles --delete-after --progress

To have the NVIDIA features available on our container centos7,  we need to install on it exactly the same drivers than in ‘local‘. Below I show the error you get if you install the wrong drivers, and a successful installation.

>./NVIDIA-Linux-x86_64-381.09.run --no-kernel-module
 Verifying archive integrity... OK
 Uncompressing NVIDIA Accelerated Graphics Driver 
for Linux-x86_64 381.09....................................
 root@centos7 ~ ## > nvidia-smi
 Failed to initialize NVML: Driver/library version mismatch
> ./NVIDIA-Linux-x86_64-375.51.run --no-kernel-module
 Verifying archive integrity... OK
 Uncompressing NVIDIA Accelerated Graphics Driver 
for Linux-x86_64 375.51....................................
 root@centos7 ~ ## > nvidia-smi
 | NVIDIA-SMI 375.51 Driver Version: 375.51 |
 | GPU Name Persistence-M| Bus-Id Disp.A    | Volatile   |
 | Fan Temp Perf Pwr:Usa | Memory-Usage     | GPU-Util C |
 | 0 GeForce GTX 1080 On | 0000:02:00.0 Off | N/A        |
 | 24% 44C P8 14W / 180W | 0MiB / 8114MiB   | 0% Default |
 | 1 GeForce GTX 1080 On | 0000:42:00.0 Off | N/A        |
 | 24% 35C P8 12W / 180W | 0MiB / 8114MiB   | 0% Default |

Of course I formatted the output of nvidia-smi to a better viewing experience. We are now able to run our software in the container. There are a few questions still open for me:

  1. Can we export this configuration to another machine, or copy it? How long does it take to do that?
  2. Can we expand the container so that it spans over several machines? Let’s say we want to have another 2 GPUs that should be seen as ‘local’. Is that feasible?

I will update this post accordingly, when I have some answers.

Posted in centos, containers, hardware, linux | Leave a comment

A bar talk

“They removed my immortality glands!” I nod. The guy closed to me started to speak like half an hour after staring at my beer in a discrete way. In the meantime, he ordered and drunk two pints of Guinness without even having a look to his display. I’m a friendly person, unfortunately, I’m not very talkative. People tend to tell me I know how to hear. “You see here?” the guy lifts his shirt on the side that is facing me. A linear scar, the size of a pen, is visible on his pale skin “they did this to me! I went to the hospital with some pain in my abdominal region, and they removed it!”

I know already most of his life: it all started with a “it’s hot here, isn’t” or something like that. Then he continued with the typical Q&A breakers: did you come here before, do you come here frequently, are you married, do you live close by, that kind of small talk. Nothing uncommon. I should have left after the standards were over, but I had nothing to do somewhere else.

“I know they did it. Before the OP, I was able to run around for hours, without any side effect. I was a good swimmer. And my skin was always having the right tone, not tanned, neither pale. Look at it now ” he waves his hands, in front of him, but not particularly showing then to me ” they are the hands of an ageing person. There are wrinkles all over. And there is the hair. ” I look at his head, trying to be as discrete as possible. He’s having a hair a little longer than usual, but nothing out of the rule. Maybe he needs a haircut or a styler. Maybe he speaks about other hair. “Each morning I find more hair in my comb. I know the hair can fall, but it was not doing it like this before! I’m no math, but at this speed, in a year I’ll be bold!” I feel on the need to interrupt before the monologue becomes gross.

“Don’t worry about it. There are adjustments that need to be done. They are expected. Sometimes they can be scary, but they are needed. You are just suffering from one.”

“Really? Since a year? I don’t think so!” He empties his third Guinness in a gulp. “I think they want to get rid of me. Yeah, that’s it. They want me to die of ageing. Or even better, they want me to commit suicide!” I look at him. Is he drunk, or simply delusional? I sip my beer to the end. The guy makes one gesture, and two new Guinness pints appear in front of us. I hate Guinness. He smiles at me. “It’s on me, man. I don’t know if I’m going to have the chance to enjoy my money for a long time.” I tentatively sip my glass. OK, it’s not as bad as usual. I check my hand display: definitely I could hold another one. “But I have a plan, my friend. I’m going to have it checked. I’m going to find out if they really removed my gland, and if they did, I’m going to sue them to their bollocks! I’m going to be the richest man in the city!”

“Yes, you are. ” I look at my hand display, then at the Guinness. I don’t know how to end this up without finishing the beer. With a discrete tap, I set up my system for a quick cleanup and I gulp the half of it. I suddenly feel drunken. “Don’t you think you’re overreacting? Maybe what you need is an update…who wants to kill a completely normal guy like you?”

“You don’t know man! I can imagine they need to select who’s staying and who’s leaving…not everybody can live forever, right?” I nod. “What if what they plan is to reduce the population, little by little, until we don’t remember how it was before? Then they can take control! They will control it all!” I gulp my beer and get my coat, on a clear attempt to leave. But I need to ask.

“Who are they?”

“The AI worshippers. The AI minions. They know everything, and they do everything his master orders. ” I’m already leaving  “Thanks for the beer man, take care” I manage to murmure. “Think about it! Think about it! ” The man is waving his beer in my direction while I leave. I try to hide in the crowd, managing only partially.

Posted in fragments, unpublished | Leave a comment

Fixing a broken RAID 50

I had that RAID 50 composed of 18 disk, each one of 6 TB, connected to my DELL server. One day particularly stressful for it, one of the disks of the RAID system decided to die.

If you are an illiterate about RAID systems, what you try first is to physically replace the broken disk. So I get my backup disk (the last 6 TB disk!) I extract the caddy with the blinking light,  unscrew the broken disk, screw the replacement, insert the caddy back and wait for a few minutes until I see that the blinking light of the disk is no more there. Also, the LED display light from the server turned from red to right, indicating everything is OK, hardware-based. Or isn’t?

Unfortunately, that was not enough. Despite there is no hardware error, the RAID is no more functional and once the OS is back online I can’t mount the share (I/O error). Time to check the RAID status. I wait until nobody is using the server, reboot it and when the time is right, I access to the RAID configuration by pressing CTRL+R.  This is what I got:


The disk I inserted is not configured, but ready. You should know what is a raid 50 file system, so I’m not going to explain it to you. I’m just going to say that, supposing that the new disk is the only problem, the fix is to add the disk to the corresponding RAID Span and rebuild the raid. Unfortunately, when I try to do that, I get to this screen:


What is going on? I did move out only one disk, why I have 2 missing? The disk count should show 18, not 16! I frantically reboot and perform a hardware scan, to try to recover the missing disk, without luck. And remember, I don’t have more 6 TB hard disks! What now?

I have no option but to destroy the virtual disk. I do so, just to find, it doesn’t matter I reboot or perform a hardware scan, I don’t detect all the 18 disks. I’m forced to go there and fiddle with the hardware. My guess is that inserting the caddy may have screwed the connection of a neighbor one, so I swap the caddy “here” with the one “there”, but still I don’t have the 18 disks online at the same time.

Running out of ideas, I take a drastic solution. I remove all the caddys, and insert them one by one (a la 2001 a Space Odyssey) pressing F5 (refresh) after each one is inserted, and I don’t stop until I have the 18 disks listed as Online. Then I make a new virtual disk, carefully trying to mimic the previous configuration (that I have written down), and when ask if I want to initiate the RAID or format the newly created virtual disk, I select NO and reboot.

System is back online. But is the data share there? Yes and no… the space is shown but I can’t mount the drive. The error message is unrecognized partition system. I could simply reformat the space, but I want to try to recover the data. Since I’m a reliable person, I google about linux data recovery before downloading and installing testdisk. Running it over the 90 TB takes quite some time, but after it, I manage to recover my partition table, and one month after the incident I can send the email to the (only) person affected by it, telling him to have a look on the lost data. Long story short, always be sure your data is backed up somewhere and you will save your time and the working time of the others. And now, I will be the first person following my advice.


Posted in bits, hardware, linux | Leave a comment

Reverse port forwarding

This is an old one. It should not be there but it is. If you don’t know what it is have a look to this instructive post. The situation is the next. We have Linux client A on site A. This one is allowed to connect via ssh to the Linux client B on site B.  However, Linux B on site B is not allowed to connect to A on site A. The reasons for this can be multiple: in my case, it is only a matter of control policies. What can we do? We’re going to enable reverse port forwarding on A, and using it to connect to A from B. I do it like this:

On client A, site A, we type:

root@A # ssh -f -N -R3322:localhost:22 -g root@

Note that A on site A, sees B with the IP What we do is we enable and dedicate port 3322 to the reverse port forwarding, start the ssh to B ( as root and send the process to the background (-N). Now we walk to site B and log in on the client, also as root. Then we type:

root@B # ssh -p 3322 root@localhost

And we should end up with a root shell from client A on site A. Caveats: you’re root. Be careful. The procedure should work with a normal user, also. I hope you’re not, anyway. Note that you can embellish the tunnel, so that you have a crontab checking the tunnel is open, and if not, it will open it, and so on. No need to write that down here, I think.

Posted in bits, centos, linux | Leave a comment

Video killed the radio star

Mann, are you still mad at us?” Maria said, with her deep voice. We walk slowly through the maze of curly streets of the historic center of our city, following Anna. Around us, some small street stands are showing vegetables and sea food. Oddly the shopkeepers are not trying to sell anything, they simply look at us, and to other like us, passing by. The sun shines. I look at Maria by my side. She is taking my hand, firmly, but maintaining a playful attitude. She smiles like one of these starts from the 50’s, Marilyn Monroe. I decide to dub her this way, since she’s also having her curves. My Marilyn.

“No. I’m not a very talkative person. On the other hand, I have a very deep personal life.” She smiles. Anna decides to intervene. “Ich hoffe that doesn’t mean that you’re all the time plotting against uns.” I send her a wary smile, then, I try to stick my tongue out of my mouth, partially for mocking, partially to test I can. I can, it’s not anymore numb. Good news. Anna ignores my gesture with a despective turn and continues walking ahead with firm steps. Her attitude remember me also another actress form the old movies, Greta Garbo. My Greta. And it’s done, I got it. I’m a radio man, walking with two video stars. And I want to break free.

Posted in aliens, dragons, fantasy, new dreams | Leave a comment