Reorder the network connections on Windows 10

connectionorder

As a part of the leginon installation we may need to reorder the network connections, so that one of them is seen always as the first one. The reordering described on the picture and link above seems to work just fine on Windows XP. However, on Windows 10 the business is a little bit different. You need to change the metrics. This is how they did it, and this is how I did it, step by step. Note that you need to be Administrator to do it.

  1. Check the original order. Open a command prompt (search Windows for it, or ask Cortana) and type ipconfig (or ipconfig /all). Keep the window open for later.
  2. Click on the network icon (usually bottom right) and on the menu that appears, on Network settings. You should get a new window. Roll down until you see Change adapter options (on blue). Click on it. You should see several icons of screens with a cable or a signal icon, maybe a red cross if the connection is not on use.
  3. Right click on the connection you want to have available first. Go to Properties (at the bottom, with the administrator shield). A new window will appear. Stay on the new window, look for  the Internet Protocol option, and click on Properties. Yet another window will appear.
  4. On the new window, click on Advanced. You should arrive to Advanced TCP/IP Settings. It has several tabs, stay on the IP settings. You see? Automatic metric is marked. Unmark  it, and give it a number above 2: for example 3. Click OK, OK, OK and close everything.
  5. Repeat for the next interface, give a number higher for the second (for example 4) and so on for the third, etc. Do it for all the interfaces! And check on the command prompt window that the order is now the one you want.

Note that you don’t need to reboot to apply these changes. And probably you were wondering why you need to give a metric above 2. It’s because 1 and 2 are reserved. You have a lot of numbers, so don’t make a movie out of this! 🙂

(EDIT: I’m kind of busy with the X-mas things. Sorry I don’t post as frequently as before…)

Advertisements

phpMyAdmin migrate database

I’m making dockers for all my web applications. And most of my web applications are hooked up with databases. And where you have a database, you have phpMyAdmin. Anyway. I have this computer A with a very extensive database that I want to have now on a docker. The docker I will call B. What we do to have the database of A in B is to dump it first.

root@A ~ ## > mysqldump -u root -p --all-databases > alldb.sql

I assume we can ssh into the docker B, and scp or transfer files to it from another location. So now we simply load the database in the docker.

[root@B ~]# mysql -u root -p < alldb.sql

That’s it. My calendars, my web, all is in B as in A. But B is a docker 🙂

A leginon docker on CentOS 7 Part II: centos7AutoInstallation.py

selinuxinstallOn the previous post, we prepared a basic LAMP docker to hold a leginon installation.  I assume now you downloaded the installer for CentOS 7 and copied it to the docker via ssh or shared folder. We can simply run it. At the beginning it looks like this:

[root@dleginon]# python centos7AutoInstallation.py 
Current OS Information: CentOS Linux release 7.3.1611 (Core)
"root" access checked success...
====================================
Installing job submission server
Installing processing server
Installing database server
Installing web server
====================================

I choose “no” on would you like to install EMAN, Xmipp, and all the processing software. We don’t want to process in the docker. Note that the auto installer seems to try to install the database server and configure it, but we already did it. Unfortunately even being careful with the options, the installer fails with multiple errors. Additional work is needed. I created a wrap over the original installer, then customize the installer so that it doesn’t throw us errors. This is my wrap:

#!/bin/bash
echo " Changing phpMyAdmin to password-protected, cookie-based "
\cp config.inc.php.phpMyAdmin.docker /etc/phpMyAdmin/config.inc.php
echo " Please setup NOW the MYSQL root password"
mysqladmin -u root password
echo " Please test login on the phpMyAdmin web interface"
echo " Setting up my.cnf limits" 
\cp my.cnf.docker /etc/my.cnf 
echo " Installing missing packages. This may take some time "
yum -y install nedit gedit net-tools torque-client torque-mom \
ImageMagick MySQL-python compat-gcc-34-g77 fftw3-devel \
gcc-c++ gcc-gfortran gcc-objc gnuplot grace gsl-devel \
libtiff-devel netpbm-progs numpy openmpi-devel opencv-python \
python-devel python-imaging python-matplotlib python-tools \
python-pip scipy wxPython xorg-x11-server-Xvfb libjpeg-devel \
zlib-devel
### to have openmpi temporary and permanently available
echo "export PATH=\$PATH:/usr/lib64/openmpi/bin" \
>> /root/.bash_profile
export PATH=$PATH:/usr/lib64/openmpi/bin
echo " ..packages installed"
### database configuration
php -f leginon-db-config.php
echo " Intial configuration done. "
echo " Starting the autoinstaller..."
python centos7AutoInstallationCustom.py
echo " Autoinstaller done. "
cp config.php.myamiweb.docker /var/www/html/myamiweb/config.php
chmod 777 /var/www/html/myamiweb
chmod 777 /var/www/html/myamiweb/config.php

As you see, we copy two template files, my.cnf.docker and config.php.myamiweb.docker. The first one is simply the one we edited on the first part of this post. The second one is not so important, since it will be overwritten after the first run of the Web Tools Setup Wizard, pictured above. Basically, just change the host name and login type.

We need to have a look to the php file also. It’s just doing what you’re asked to do for the Database Server Installation. Let’s say your docker is called “dleginon“. My php file is like this:

<?php 
 $connection = mysql_connect( "127.0.0.1", "root", "docker" ) 
 or die( "Sorry - unable to connect to MySQL" );
 echo( " Congratulations - you are connected to MySQL \n" );
 echo( " Creating initial databases and users \n" );

mysql_query("CREATE DATABASE leginondb");
mysql_query("CREATE DATABASE projectdb");

mysql_query("CREATE USER 'usr_object'@'localhost' \
IDENTIFIED BY 'DOCKERLEGINON';");
mysql_query("GRANT ALL PRIVILEGES ON leginondb.* TO \
'usr_object'@'localhost';");
mysql_query("GRANT ALL PRIVILEGES ON projectdb.* TO \
'usr_object'@'localhost'docker;");

mysql_query("GRANT ALL ON leginondb.* TO \
'user_object'@'localhost'");
mysql_query("GRANT ALL ON projectdb.* TO \
'user_object'@'localhost'");
mysql_query("GRANT ALTER, CREATE, INSERT, SELECT, UPDATE, \
DELETE ON ON `ap%`.* TO 'user_object'@'localhost'");
mysql_query("GRANT ALL PRIVILEGES ON *.* TO \
'usr_object'@'localhost'");
 
mysql_query("CREATE USER 'usr_object'@'%' \
IDENTIFIED BY 'DOCKERLEGINON';");
mysql_query("GRANT ALL PRIVILEGES ON leginondb.* TO \
'usr_object'@'%';");
mysql_query("GRANT ALL PRIVILEGES ON projectdb.* TO \
'usr_object'@'%';");

mysql_query("GRANT ALL ON leginondb.* TO 'user_object'@'%'");
mysql_query("GRANT ALL ON projectdb.* TO 'user_object'@'%'");
 
mysql_query("GRANT ALTER, CREATE, INSERT, SELECT, UPDATE, \
DELETE ON ON `ap%`.* TO 'user_object'@'%'");
mysql_query("GRANT ALL PRIVILEGES ON *.* TO 'usr_object'@'%'");

mysql_close();
 echo( " ...done \n" );

?>

The user “usr_object” is having the password “DOCKERLEGINON“, while the root docker is having the password “docker“. Be aware that inconsistency on root and  usr_object users on the database will lead to errors. Now what is centos7AutoInstallationCustom? I will not post it here (it’s too long) but basically it’s the auto installer you downloaded minus the system calls and minus the last lines that are opening the browser. I will not post it here, just modify it yourself.

Now we need to run the Web Tools Setup Wizard.  If your docker is called dleginon, open your browser pointing to http://dleginon/myamiweb/setup/ and use Database User: usr_object, Database password: DOCKERLEGINON. It is recommended to restart the docker first, in case you didn’t do it. Once you’re done, you should be able to log in as Administrator onto the myamiweb.

Are we ready to go? Not yet. If you ssh into the container to run start-leginon.py, you will get the next error:

leginon.project.NotConnectedError: 
(1045, "Access denied for user 'root'@'localhost' 
(using password: NO)")

This is due to an unconfigured /etc/myami/sinedon.cfg. Edit the file, and change the username and password so that start-leginon can access the database. If you run it afterwards, you will get this error:

RuntimeError: Must create at least one project 
before starting Leginon

To get rid of this one, go to myamiweb, and create a test project. Then run start-leginon.py again. Now it should work, and if not, let me know 🙂

A docker macvlan on CentOS 7

I want to dockerize my services (not only leginon) so I need to find out an effective way to integrate them onto our intranet. That is proven to be a little bit complex, since I have partial control of our intranet. Here I explain how to create a secure docker network, where each docker can see the other, but they can’t see outside world.

First we need to stop the docker service, and edit the daemon.json docker configuration file on /etc/docker. In my case, it is empty. We fill it with a bip and fixed cidr. Then we need to successfully restart the docker daemon. Here you have a complete example of a docker bridge file definition.

{
  "bip": "192.168.1.5/24",
  "fixed-cidr": "192.168.1.5/25",
  "fixed-cidr-v6": "2001:db8::/64",
  "mtu": 1500,
  "default-gateway": "10.20.1.1",
  "default-gateway-v6": "2001:db8:abcd::89",
  "dns": ["10.20.1.2","10.20.1.3"]
}

I used  only the parameters needed. If your bip or fixed-cidr are wrong, the docker daemon will not start. Now I create my docker network mynet with a command I took from the docker overlay documentation.

docker network create -d macvlan \
--subnet=MY.SUBNET.IS.0/24 \
--ip-range=MY.SUBNET.IS.1/24 \
--gateway MY.SUBNET.IS.1 \
-o parent=vlan-TEST mynet

The output should be the typical string of letters and numbers. If your parameters are wrong, you can get this meaningless errors instead:

Error response from daemon: 
Pool overlaps with other one on this address space

or

no matching subnet for gateway MY.SUBNET.IS.1

Just try until it works. We create two dummy containers (I will let you choose the names, mines are leginon1 and leginon2) and we check their isolated IPs:

docker network inspect mynet | grep -i ipv4

Note that the alias that is giving us the IP of the docker will not work because they are isolated. Also I didn’t find something similar for network inspect, so grep it is. If you want to see all the info from the JSON file, just remove the grep part. Let’s log in each one and ping the other from the other like this:

shell1 > docker exec -i -t leginon1 /bin/bash
shell2 > docker exec -i -t leginon2 /bin/bash

There you have it! Isolated containers communicating with each other.
I quote from here: Linux Macvlan interface types are not able to ping or communicate with the default namespace IP address. For example, if you create a container and try to ping the Docker host’s eth0 it will not work. That traffic is explicitly filtered by the kernel to offer additional provider isolation and security. That means also that an alias that can give us the IP of the docker will not to work.

So don’t do this if you want to have dockers available from other machines different from the one running the docker daemon. But if you did, maybe now you want to remove the traces. It’s very simple.

  1. Stop the daemon  # > systemctl stop docker
  2. Remove the network information, by editing the daemon.json file to an empty state (just the two brackets)
  3. Delete the docker and the bridge using ip link del as explained here.
  4. Start the daemon

And now, back to the drawing board 😦

A leginon docker on CentOS 7 Part I: docker and database server install

Previously we we made a leginon CentOS 6 container on CentOS 7. Now the LXC containers are somehow more robust than a docker, but less popular and versatile. Therefore the need to make a docker instead. I start by cloning the CentOS 7 docker LAMP solution.

git clone https://github.com/wrender/centos-docker-lamp.git

The above command will create a folder called centos-docker-lamp. Inside you have the basic CentOS 7 LAMP (Linux-Apache-MySql-PhP) Dockerfile. Before making one, on the docker host, I will define a virtual interface with the IP that I want to give to my docker, in this case, 1.234.56.123 (not real). I will also make a script to start a docker the way I like it, baby. This is my script:

#!/bin/bash
docker run --name dleginon \
 --hostname=dleginon \
 -d -p 1.234.56.123:80:80 \
 -p 1.234.56.123:443:443 \
 -p 1.234.56.123:2222:22 \
 -v `pwd`/html:/var/www/html \
 -v `pwd`/database:/var/lib/phpMyAdmin/upload \
 -t otherdata/centos-docker-lamp:5.6

I first start the interface with the docker ip, and then I run the docker.  If you don’t have the interface up, you will get an error like this:

/usr/bin/docker-current: Error response from daemon: 
driver failed programming external connectivity on endpoint 
dleginon (contanier-long-list-of-numbers-and-letters): 
Error starting userland proxy: listen tcp 1.234.56.123:443: 
bind: cannot assign requested address.

Note that I get the image for php 5.6. This is important, otherwise our installation will fail, due to the changes on the php functions we will get an undefined mysql connect error. Please notice also that I map all the docker’s service ports to the hosts ports except the ssh. If we don’t get an error after running the script, we will see a dleginon container listed on docker ps -a. I have a DNS entry as dleginon for the IP, so I can ssh in easily. I do it.

ssh -Y root@dbleginon -p 2222

The root password is defined inside the Dockerfile. But we can change it now, using passwd. Let’s check what we got:

root@dbleginon ~]# cat /etc/*-release
CentOS Linux release 7.3.1611 (Core) 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.3.1611 (Core) 
CentOS Linux release 7.3.1611 (Core)

Not bad. We can go to the phpmyadmin page by typing on a browser: http://dleginon/phpMyAdmin/ or the IP address of dleginon,  http://1.234.56.123/phpMyAdmin/. We do get the phpMyAdmin interface, but “unprotected”. Let’s leave it like that for the moment, and start creating the database leginon configuration. In the dleginon container, we type:

[root@dleginon ~]# mysqladmin create leginondb
[root@dleginon ~]# mysqladmin create projectdb

Steps 7,8,9, and 10 of the database configuration guide above don’t give troubles. Note that we create usr_object for “localhost” and for “domain” (mydomain.edu) for convenience. We change the root password and we test the query as suggested. Step 13, however, doesn’t look right. We forgot to set some limits. We change them now.

Edit /etc/my.cnf to add or change query cache variables:

# The MySQL server
[mysqld]
query_cache_type = 1
query_cache_size = 100M
query_cache_limit= 100M

In order to protect phpMyAdmin, I copy on /etc/phpMyAdmin/ the config.inc.php from the “real” leginon server. I didn’t copy /etc/my.cnf, since the “real” one was looking quite more complex than the one of the container.

Now we need to stop the container and restart it again, or the changes will not take effect. We need to remember a docker is not a system, so systemd calls will produce a D-BUS error. No problem if everything works at the end. And it does. Once I stop and start the container, I get my interface protected by the “new” root password. And the user_object printout looks fine:

MariaDB [leginondb]> SHOW VARIABLES LIKE 'query%';
+------------------------------+-----------+
| Variable_name                | Value     |
+------------------------------+-----------+
| query_alloc_block_size       | 16384     |
| query_cache_limit            | 104857600 |
| query_cache_min_res_unit     | 4096      |
| query_cache_size             | 104857600 |
| query_cache_strip_comments   | OFF       |
| query_cache_type             | ON        |
| query_cache_wlock_invalidate | OFF       |
| query_prealloc_size          | 24576     |
+------------------------------+-----------+

What now? We test PHP can connect:

[root@dleginon ~]# php -r "mysql_connect('localhost', 
'usr_object', 'MY-SPECIAL-PASSWORD', 'leginondb'); 
echo mysql_stat();"; echo "" 
Uptime: 282 Threads: 1 Questions: 7 Slow queries: 0 Opens: 
19 Flush tables: 1 Open tables: 11 Queries per second avg: 0.024

And we leave the rest for the next post.

Windows XP ignoring the hosts file

A little bit of context is needed. We have a subnet with 3 computers that need to connect to each other. It’s a Windows XP, a Windows 7 and a Centos 6. We can’t get rid of the OS, since they are controlling devices. We need to be able to ping by name, and unfortunately, we can’t hook them up with a DNS server.

We are doing this to be able to run leginon on the subnet. The installation guide is telling us how to add the hosts so that each member is able to see each other. Unfortunately, on the Windows XP, no matter what we write on the host file, we can’t ping the others by name. If you google the problem, a lot of nice information is coming back. We tried what is explained on this thread on serverfault without success. First we check the that the Registry Key is pointing to the right entry. We open regedit, and look for:

The key that is on

\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\
Services\Tcpip\Parameters\DataBasePath

is indeed:

%SystemRoot%\System32\drivers\etc

So I assume is not the key. To rule out permission problems, we cut the host file out and paste it on the desktop, we create a new one and fill it up, without luck. Also the right click path (Properties->Security) is looking OK. All the users are having all the relevant permissions. We then change the proxy settings by going to Internet Explorer -> Internet Options -> Connections -> LAN settings. Also it look like is suggested. Just in case, we check the network configuration, but still no ping. We can’t flush the DNS cache (on a cmd window, ifconfig /flusdns), since we don’t have a DNS on the subnet. What is it?

I found the answer here. I go to Start–>Control Panel–>Administrative Tools–>Component Services–>Services(Local).

It looks like this, depending on your XP tuning:servicesdialog

I search for the service called DNS Cache (not in the picture) and disable by clicking on it. Then I come back to my host file, test the ping by name and voila! It works! Finally I can ping from the Windows XP. Why the DNS cache was configured like that, I will never know. Why I didn’t think about it in the first place, I can explain, if asked. Anyway, thanks google, for being there when I need you 😀

A leginon CentOS 6 container on CentOS 7

Continuing with our cooking recipes, I’m going to install leginon on a CentOS 6 container running on CentOS 7. We start creating a CentOS 6 container:

lxc-create -n centos6 -t download -- -d centos -r 6 -a i386
lxc-start -n centos6

To log in via ssh we need first to install the openssh server. Also, let’s install more things that we will need if we want to have a nice leginon installation experience.

yum install -y git which openssh-server wget \
unzip tar ffmpeg ffmpeg-devel

Now we better restart the container and log in as root via ssh. We’re ready to start installing. Since we have CentOS 6 we can directly use the autoinstaller.  I’m not going to discuss the previous steps of the complete installation but go to step 6 directly. So we do

python centosAutoInstallation.py

This is retrieving and installing every package we need using different tools, so we need to monitor everything goes smoothly. In my case, the first time it didn’t finish successfully due to several packages missing. I will blame my basic container for that. I choose no sample download, no software install, and I get:

ERROR: Failed to run Myamiweb setup script.
You may try running http://localhost-something 
in your web browser.

Let’s try to go ahead and ignore the error. Next issue occurs when I try to run firefox from inside the container.

firefox
process 16203: D-Bus library appears to be incorrectly set up; 
failed to read machine uuid: 
Failed to open "/var/lib/dbus/machine-id": 
No such file or directory
See the manual page for dbus-uuidgen to correct this issue.
 D-Bus not built with -rdynamic so unable to print a backtrace
Redirecting call to abort() to mozalloc_abort
Segmentation fault

The solution to the D-Bus library problem is easy.

> dbus-uuidgen > /var/lib/dbus/machine-id

I open firefox and I check that I can see the leginon start page. This is not the end, now we need to do the database server installation.  I cut and copy from the HOWTO:

# yum install mysql mysql-server # already done by autoinstaller  
ls /usr/share/mysql/my* --> they are indeed there
cp -v /usr/share/mysql/my-huge.cnf /etc/my.cnf

I modify my.cnf as suggested, I add  default_storage_engine=MyISAM at the end of the file, and I service mysqld start, chkconfig mysqld on. So far so good.

Since we want to have a container as portable thing, instead of creating the database, we are going to import it. We do as I wrote on phpMyAdmin upload and fix old database plus an additional step:

chown -R mysql /var/lib/mysql/

Otherwise starting mysqld fails with the error:

[ERROR] Can't open and lock privilege tables: 
Table 'mysql.servers' doesn't exist

We can open phpMyAdmin on firefox now, or do the checks suggested on step 13 and 14. All looks clear. What now? We do have already a leginon server up and running on ‘real’ hardware , so instead of  runnig the online setup wizard by visiting http://yourhost/myamiweb/setup or http://localhost/myamiweb/setup to create the myami website’s config file, as described here, we simply rsync the folder from the running one to our container, and perform service httpd restart. Provided the data folder is mounted in the container, and so on, we now have a working leginon container. In principle we can stop here, but we want to update it regularly so that we have the latest software version. The first update I do manually. First we clone the beta version of myami like this:

git clone -b myami-beta http://emg.nysbc.org/git/myami myami-beta
cd /path/myami-beta
./pysetup.sh install

And we check that everything works:

~/myami-beta/leginon ## > python syscheck.py
 --------------------------------------------------------------
 Looking for previously installed Leginon...
 Leginon found here: /usr/lib/python2.6/site-packages/leginon
 *** It is best to uninstall your previous 
Leginon before installing the new one. 
The best way to uninstall is to move it to a backup
location, just in case you need to revert to the old version.
--------------------------------
 Python:
 Python executable (if wrong, check PATH in your environment):
 /usr/bin/python
 Python module search path (if wrong, check PYTHONPATH):
 /root/myami-beta/leginon
 /usr/local/protomo-2.4.1/lib/linux/x86-64
 /usr/local/EMAN/lib
 /usr/lib/python26.zip
 /usr/lib/python2.6
 /usr/lib/python2.6/plat-linux2
 /usr/lib/python2.6/lib-tk
 /usr/lib/python2.6/lib-old
 /usr/lib/python2.6/lib-dynload
 /usr/lib/python2.6/site-packages
 /usr/lib/python2.6/site-packages/PIL
 /usr/lib/python2.6/site-packages/
setuptools-0.6c11-py2.6.egg-info
 /usr/lib/python2.6/site-packages/wx-2.8-gtk2-unicode
 Python version: 2.6.6
 OK (at least 2.3.4 required)
 Python says home directory is: /root
 --------------------------------------------------------------
 Python Imaging Library (PIL):
 importing Image module...
 PIL version: 1.1.6
 OK (at least 1.1.4 required)
 -------------------------------------------------
 MySQL Python client (MySQLdb):
 importing MySQLdb module...
 Python MySQL version: 1.2.3c1
 OK (at least 1.2 required)
 -------------------------------------------------
 numpy:
 importing numpy module...
 numpy version: 1.4.1
 OK (at least 1.0 required)
 --------------------------------------------------
 scipy:
 importing scipy.optimize module...
 testing for leastsq function...
 OK
 -------------------------------------------------
 wxPython:
 importing wx module...
 wxPython version: 2.8.12.0
 OK (at least 2.5.2.8 required)
 Testing a wxPython application. 
Close the window that pops up...
 TEST
 wxPython test successful

So it seems to work. We could copy the stuff from myami-beta to the location given on the test,  /usr/lib/python2.6/site-packages/leginon. Time for the power user to test it! We’ll see how long until we screw it up 🙂