ERROR : /usr/share/Modules/init/sh: No such file or directory on Ubuntu 22.04

We started the migration to a new Linux flavour – in principle Ubuntu– and therefore we started also to experience the first issues. We have a very established software module method, with the module definitions being loaded from a network location. We can install software modules, but after every folder needed and every other adjustment is made we see this:

$ > ssh user@new-server
Last login: DATE from other-server.org
-bash: /usr/share/Modules/init/sh: No such file or directory

I have a machine “other-server” to check how the modules are working. And I find that the folder “Modules” is missing in Ubuntu. The content of the folder modules in the same location seems to be the same, so I do

root@new-server:/usr/share# ln -s /usr/share/modules Modules

And try logging again. With this result:

$ > ssh user@new-server
user@new-server's password: 
Last login: DATE from other-server.org
user@new-server ~ $ > module avail
/PATH/modulecmd: 
error while loading shared libraries: 
libtcl8.5.so: cannot open shared object file: 
No such file or directory

So there’s now a library problem. Let’s try to fix it very quickly. I localize a similar library

root@new-server:~# ls /usr/lib/*/*libtcl*
/usr/lib/x86_64-linux-gnu/libtcl8.6.so 
/usr/lib/x86_64-linux-gnu/libtcl8.6.so.0 
/usr/lib/x86_64-linux-gnu/libtclenvmodules.so
root@new-server:~# 
cp /usr/lib/x86_64-linux-gnu/libtcl8.6.so /usr/lib/libtcl8.5.so
root@new-server:~# ldconfig

And the error is gone. We can do this because the libraries are not so different…. I have no idea of the side effects. Let’s hope there are not so many 😉!

Advertisement

ModuleNotFoundError: No module named ‘absl’ while using python

This is a very specific error. We have quite a mess-up python setup with multiple versions, network modules and local, a SLURM cluster, and the option to install your own, so it’s quite tricky to track the origin of a module. But it helps if I have two servers with the same kernel and packages, one of them the program runs, but not in the other. I do have logs, but they are not very clear. Message says

ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'python-3.7.3'
Traceback (most recent call last):
File "/XXX/run_docker.py", line 22, in <module>
from absl import app
ModuleNotFoundError: No module named 'absl'

So what is happening here I believe is that the program is trying to load the module, it fails and goes to the local install. As an user I can install the missing absl-py module, but I can’t run it in this case because of the special process. What to do then? All the solutions on stackoverflow were not suitable, since I don’t know which python is getting what. But I have a hint: the program seems to be loading python 3.7.3, that is not the default. So I look for the python modules stored and I found them on the server one. A simple sync

@ server-one ## > rsync -av /usr/local/lib/python3.6/site-packages/ root@server-two:/usr/local/lib/python3.6/site-packages/

and then the “program” works. What is going on? It looks like python, since it didn’t manage to find the modules for the requested python version, took the closest ones available (3.6). But who knows? I’m not a python expert, I’m just passing by 😔.

ModuleNotFoundError: No module named ‘packaging’ on CentOS 7.X

We have this new version of EMAN installed as a module since quite some time in our SLURM cluster, but oddly enough until today no one reported that the GUI was not opening when performing SSH to one of our big servers. Specifically, when performing ssh from a mac. This is how the error looks like (with the irrelevant areas obscured as usual):

~ $ > module load eman-2.91 
~ $ > e2projectmanager.py 
Traceback (most recent call last):
File "/EMAN2/bin/e2projectmanager.py", 
line 40, in <module>
from eman2_gui.empmwidgets import *
File "EMAN2/lib/python3.7/
site-packages/eman2_gui/empmwidgets.py", 
line 44, in <module>
from .emselector import EMSelectorDialog 
# This will be replaced by something more sensible in the future
File "EMAN2/lib/python3.7/site-packages/eman2_gui/emselector.py", 
line 50, in <module>
from .emplot2d import EMPlot2DWidget
File "EMAN2/lib/python3.7/site-packages/eman2_gui/emplot2d.py", 
line 83, in <module>
import matplotlib.pyplot as plt
File ".local/lib/python3.7/site-packages/matplotlib/__init__.py", 
line 105, in <module>
from packaging.version import parse as parse_version 
ModuleNotFoundError: No module named 'packaging'

Well it looks obviously a python error. I found the relevant StackOverflow post for no module called packaging and perform the pip3 install packaging. This is my output, for the records:

$ > pip3 install packaging
Defaulting to user installation because 
normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, 
https://pypi.ngc.nvidia.com
Collecting packaging
Downloading packaging-21.3-py3-none-any.whl (40 kB)
━━━━━━━━━━━━━━━━━━ 40.8/40.8 KB 54.3 MB/s eta 0:00:00
Requirement already satisfied: 
pyparsing!=3.0.5,>=2.0.2 in 
EMAN2/lib/python3.7/site-packages (from packaging) (3.0.4)
Installing collected packages: packaging
ERROR: pip's dependency resolver does not currently 
take into account all the packages that are installed. 
This behaviour is the source of the following dependency conflicts.
matplotlib 3.5.1 requires pillow>=6.2.0, 
which is not installed.
Successfully installed packaging-21.3
WARNING: You are using pip version 22.0.4; 
however, version 22.1.2 is available.
You should consider upgrading via the 
'EMAN2/bin/python -m pip install --upgrade pip' command.

Interesting 🤔 . You get an error but it works. Unfortunately that is not enough to get the wanted GUI. When I try to call it again, another error pops up. This time it reads:

ModuleNotFoundError: No module named 'PIL'

Again StackOverflow has the right answer. We install Pillow, that is the heir of PIL, like this:

$ > pip install Pillow

And now our GUI pops up. Ah, python, python. Like the beer, 🍺 , dear python, you are the source and the solution to all my problems! 😁😁😁.

Install openmpi-4.1.2 on CentOS 7.X

Previously I had a few issues installing openmpi on CentOS. Then I was relieved of the software installation task and out of this business of module creation for a while. Now that I’m back I’m gladly surprised that the installation of openmpi-4.1.2 as a module happened smoothly. This is what I did.

1. Donwload the tar ball from the official site.
2. Untarr and configure
3. make it

You can go to my previous post, but for the configuration and install I run

## > tar -xvzf openmpi-XXX.tar.gz
## > ./configure --prefix=/path/to/openmpi-XXX 
--with-cuda=/path/to/cuda-XXX/
## > make all install

The original mpi module also I didn’t modify so much, just I updated the path to the new install, and changed the version number. If everything were so easy as this… but it’s not. 😔.

Compiling SIDESPLITTER v1.2 on CentOS 7.X

Today’s working log is about SIDESPLITTER, a Local SNR Filter for both SIDES of SPLIT refinements. If you don’t know what that means, maybe you should study more 😉

The instructions on the github page here are a little meager but they were proven correct. On a few easy steps that I can perform as an user, first I clone the repo…

git clone https://github.com/StructuralBiology-ICLMedicine/SIDESPLITTER.git

Then I go into the newly created folder (SIDESPLITTER) and try

mkdir build && cd build
cmake ..

That produces an error in my case that reads

CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
  CMake 3.1 or higher is required.  You are running version 2.8.12.2
-- Configuring incomplete, errors occurred!

Fortunately we have a module with a modern cmake. We load the module and try again. This time we have an error with the FFTW libraries (Fast Fourier Transform libraries). Like this:

CMake Error at CMakeLists.txt:14 (find_package):
  Could not find a package configuration file provided by "FFTW3" with any of
  the following names:

    FFTW3Config.cmake
    fftw3-config.cmake

  Add the installation prefix of "FFTW3" to CMAKE_PREFIX_PATH or set
  "FFTW3_DIR" to a directory containing one of the above files.  If "FFTW3"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!

I have the libraries and I can link them, but instead let’s do as you read when you click on the github link. I quote the message just in case:

N.B. IF YOU EXPERIENCE FFTW LIBRARY ERRORS WITH THE CMAKE FILE, PLEASE
USE THE COMPILE.SH BASH SCRIPT PROVIDED – REQUIRES FFTW_DEVEL LIBRARIES

So I make the script executable (chmod 777 compile.sh) and run it. This will create a new binary sidesplitter that we can directly run to check the version and enjoy the ASCII art coming with it 🙂

error while loading shared libraries: libtcl8.5.so: cannot open shared object file: No such file or directory on CentOS 8

Another error with the new install. This one appeared just after module avail to check if the network-mounted modules for the old distro (CentOS 7.8) work fine on CentOS 8. Upon TCL install I got /usr/lib64/libtcl8.6.so, and later on I installed anaconda3, that was coming with their own versions of the libraries. So my find output looks like this:

root@test ~ ## > find / -name *libtcl8*
/root/anaconda3/pkgs/tk-8.6.10-hbc83047_0/lib/libtcl8.6.so
/root/anaconda3/lib/libtcl8.6.so
/usr/local/anaconda3/pkgs/tk-8.6.10-hbc83047_0/lib/libtcl8.6.so
/usr/local/anaconda3/lib/libtcl8.6.so
/usr/lib64/libtcl8.6.so

I have tried yum remove/yum install tcl8.5\* and yum install tcl-devel out of a hunch without luck. Here you have how TCL is installed on CentOS 7, if you need it. I’ve used it as a reference. Desperate and not wanting to modify drastically the system, I found a solution. On the tcl/tk version reference it is written, as expected, that Tcl 8.6 is highly compatible with previous Tcl versions. So I just simply did this:

root@test ~/usr/lib64 ## > cp libtcl8.6.so libtcl8.5.so
root@test ~/usr/lib64 ## > ll libtcl*
-r-xr-xr-x 1 root root 1953008 Sep 15 10:24 libtcl8.5.so*
-r-xr-xr-x 1 root root 1953008 May 11 2019 libtcl8.6.so*
lrwxrwxrwx 1 root root 12 May 11 2019 libtcl.so -> libtcl8.6.so*
-rwxr-xr-x 1 root root 15948 May 11 2019 libtclstub8.6.a*
root@test ~/usr/lib64 ## > ldconfig

Log out, log in again and the error is gone. I can list the modules available. Now, time to test if they work. One by one, I’m afraid. Wish me luck! 😦

Installing python 3.8.0 as a module on CentOS 7

I get the package from here. Unzip it on my test folder, move it to my software folder called /net/ here, and do as suggested on tecmint. I add to the commands in blue a little output with comments to understand it better.

# tar xJf Python-3.6.3.tar.xz
# cd Python-3.6.3
# ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
... a lot of checks here ...
configure: creating ./config.status
config.status: creating Makefile.pre
config.status: creating Misc/python.pc
config.status: creating Misc/python-embed.pc
config.status: creating Misc/python-config.sh
config.status: creating Modules/ld_so_aix
config.status: creating pyconfig.h
creating Modules/Setup.local
creating Makefile
If you want a release build with all stable 
optimizations active (PGO, etc),
please run ./configure --enable-optimizations
# make
gcc -pthread -c -Wno-unused-result -Wsign-compare 
-DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra 
-Wno-unused-result -Wno-unused-parameter 
-Wno-missing-field-initializers 
-Werror=implicit-function-declaration 
-I./Include/internal -I. -I./Include -DPy_BUILD_CORE 
-o Programs/python.o ./Programs/python.c
... and a lot of lines like the one above...
gcc -pthread -Xlinker -export-dynamic 
-o Programs/_testembed Programs/_testembed.o libpython3.8.a 
-lcrypt -lpthread -ldl -lutil -lm -lm 
sed -e "s,@EXENAME@,/usr/local/bin/python3.8," < 
./Misc/python-config.in >python-config.py
LC_ALL=C sed -e 's,\$(\([A-Za-z0-9_]*\)),\$\{\1\},g' 
< Misc/python-config.sh >python-config
# make install
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-19.2.3 setuptools-41.2.0

End of the dump. I have compiled it on /net/python-3.8.0/ without any further configuration options. After make install, I still don’t have the new python by default, but I’m able to run it:

root@test ~ ## > /net/python-3.8.0/python 
Python 3.8.0 (default, Nov 7 2019, 16:17:32) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" 
for more information.
>>> exit()

Let’s make a module for it:

#%Module1.0##############################
## modules python-3.8.0
## modulefiles/python-3.8.0 . Python 3.8.0 module
##
proc ModulesHelp { } {
   global version modroot
   puts stderr "setenv python3.8.0"
}
module-whatis "Sets the environment for python-3.5.3"
# for Tcl script use only
set topdir /net/python-3.8.0
set version 3.8.0
set sys linux86
setenv PYTHON_V "3.8.0"
prepend-path PATH $topdir

Now I can do module load python-3.8.0 and I will have my brand new python. Next step: install the requested “modern” python code. The output, maybe tomorrow 🙂

EDIT: Althought it’s a functional python 3.8.0 prompt, it lacks conda. Trying to install JANNI  with conda, the next message is displayed:

conda create -n janni -c anaconda python=3.6 cudnn=7.1.2 
libtiff wxPython=4.0.4
ERROR: The install method you used for conda--probably 
either `pip install conda` or `easy_install conda`
--is not compatible with using conda as an application.
If your intention is to install conda as a standalone 
application, currently supported install methods 
include the Anaconda installer 
and the miniconda installer. 
You can download the miniconda installer from
https://conda.io/miniconda.html.

So back to square zero 😦

/lib64/libstdc++.so.6: version `CXXABI_1.3.8′ not found on CentOS 7

This error appears when you run one module over another. The modules configure the environmental variables as PATH and LD_LIBRARY_PATH. So it’s easy to screw up the path, and as far as I know, there’s no easy way to come back to a standard env.  This is the stack overflow solution. The module that throws the error in my case is ChimeraX. Here’s what I did.

user@term ~ $ > module load chimeraX
user@term ~ $ > LD_LIBRARY_PATH=/usr/local/lib64/:$LD_LIBRARY_PATH
user@term ~ $ > export LD_LIBRARY_PATH
user@term ~ $ > ChimeraX

We have moved the software to a network repository, but not the “system” libraries (like lib64) so sometimes you need to add the local ones as well as the ones running with the network repository.

CryoSPARC 2 slurm cluster worker update error

This is about CryoSPARC again. Previously we did install it on CentOS and update it, but on a master + node configuration, not on a cluster configuration. If it’s a new install on your slurm cluster, you should follow the master  installation guide, that tells you to make a master install on the login node, then, on the same login node install the worker:

module load cuda-XX
cd cryosparc2_worker
./install.sh --license $LICENSE_ID --cudapath 

The situation is that we update the master node but the Lane default (cluster) doesn’t get the update and the jobs crash because of it. First we uninstall the worker using one of the management tools like this:

cryosparcm cli 'remove_scheduler_target_node("cluster")'

Then we cryosparc stop and we move the old worker software folder

mv cryosparc2_worker cryosparc2_worker_old

and get a new copy of the worker software with curl.

curl -L https://get.cryosparc.com/\
download/worker-latest/$LICENSE_ID \ > cryosparc2_worker.tar.gz

We cryosparc start, and untar, cd, and install. Don’t forget to add your LICENSE_ID and to load the cuda module or be sure you have one cuda by default. This is an edited extract of my worker install:

******* CRYOSPARC SYSTEM: WORKER INSTALLER ***********************

Installation Settings:
License ID :  XXXX
Root Directory : /XXX/Software/Cryosparc/cryosparc2_worker
Standalone Installation : false
Version : v2.5.0

******************************************************************

CUDA check..
Found nvidia-smi at /usr/bin/nvidia-smi

CUDA Path was provided as /XXX/cuda/9.1.85
Checking CUDA installation...
Found nvcc at /XXX/cuda/9.1.85/bin/nvcc
The above cuda installation will be used but can be changed later.

***********************************************************

Setting up hard-coded config.sh environment variables

***********************************************************

Installing all dependencies.

Checking dependencies... 
Dependencies for python have changed - reinstalling...
---------------------------------------------------------
Installing anaconda python...
----------------------------------------------------------
PREFIX=/XXX/Software/Cryosparc/cryosparc2_worker/deps/anaconda
installing: python-2.7.14-h1571d57_29 ...

...anaconda being installed...
installation finished.
---------------------------------------------------------
Done.
anaconda python installation successful.
---------------------------------------------------------
Preparing to install all conda packages...
-----------------------------------------------------------
----------------------------------------------------------
Done.
conda packages installation successful.
------------------------------------------------------
Preparing to install all pip packages...
----------------------------------------------------------
Processing ./XXX/pip_packages/Flask-JSONRPC-0.3.1.tar.gz

Running setup.py install for pluggy ... done
Successfully installed Flask-JSONRPC-0.3.1 
Flask-PyMongo-0.5.1 libtiff-0.4.2 pluggy-0.6.0 
pycuda-2018.1.1 scikit-cuda-0.5.2
You are using pip version 9.0.1, 
however version 19.1.1 is available.
You should consider upgrading via the
 'pip install --upgrade pip' command.
-------------------------------------------------------
Done.
pip packages installation successful.
-------------------------------------------------------
Main dependency installation completed. Continuing...
-------------------------------------------------------
Completed.
Currently checking hash for ctffind
Dependencies for ctffind have changed - reinstalling...
--------------------------------------------------------
ctffind 4.1.10 installation successful.
--------------------------------------------------------
Completed.
Currently checking hash for gctf
Dependencies for gctf have changed - reinstalling...
-------------------------------------------------------
Gctf v1.06 installation successful.
-----------------------------------------------------------
Completed.
Completed dependency check.

******* CRYOSPARC WORKER INSTALLATION COMPLETE *****************

In order to run processing jobs, you will need to connect this
worker to a cryoSPARC master.

****************************************************************

We are adding a worker that was somehow previously there, so I don’t do anything else. If I check the web, Lane default (cluster) is back. Extra tip: the forum entries about a wrong default cluster_script.sh, and about the Slurm settings for cryosparc v2.

If I need to add something: be aware that the worker install looks like coming with its own python, and it does reinstall cftfind and gctf. So be careful if you run python things in addition to cryosparc 🙂

Install EMAN2.2 with cmake 3.13 on CentOS 7

I’m not going to bloat the post with personal opinions about how good is the installation method is. Here you get the EMAN2 sources. This is the HOWTO install , and from there I did the compilation with anaconda. As you may know from previous posts, we use modules. First we load our python module and upgrade conda like suggested on the error message, if you get it. Then we go ahead with the instructions. I do that on a new window.

module load python-2.7.13
conda install eman-deps=13 -c cryoem -c defaults -c conda-forge

In my case, 27 new packages are installed, 14 packages are updated, 16 packages are downgraded, meaning around 40 packages are downloaded and extracted. Since we do that over the python modul, (I hope) I do need to do this only once. It takes like 10 minutes to be done, but multiply that by 30!  Now I cmake what I got from github. As expected, I get a cmake version error.

eman22-build ## > 
cmake ../eman22-src/ -DENABLE_OPTIMIZE_MACHINE=ON
CMake Error at CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
CMake 3.9 or higher is required. You are running version 3.8.2

-- Configuring incomplete, errors occurred!

Let’s donwload then a new cmake and make a module for it. I get the binary cmake for my linux, the cmake-3.13.3-Linux-x86_64.tar.gz, so I don’t need to compile it. The unzipped folder I place on /network/cmake-3.13.3/. My module looks like this then:

## modules cmake-3.13.3
## modulefiles/cmake-3.13.3. Sample gcc module
proc ModulesHelp { } {
global version modroot
   puts stderr "cmake-3.13.3"
}
module-whatis "Sets the environment for using cmake-3.13.3"

# for Tcl script use only
set topdir /network/cmake-3.13.3
set version 3.13.3
set sys linux86

setenv CMAKE_V "3.13.3"

prepend-path PATH $topdir/bin
prepend-path MANPATH $topdir/man
prepend-path LD_LIBRARY_PATH $topdir/lib

Time to cmake again. I do it on a new terminal. First I load both modules, my python and my cmake. Then I just follow the instructions…

cmake ../eman22-src/ -DENABLE_OPTIMIZE_MACHINE=ON

And make -j 4, make install. OK! The final module (eman22 installed on /network/eman22-build) looks like this:

#################
## modules eman22-2019.1
## modulefiles/eman22-2019.1 EMAN module
##
proc ModulesHelp { } {
global version modroot
   puts stderr "eman22-2019.1"
}
module-whatis "EMAN 22 2019.1"
# for Tcl script use only
set topdir /network/eman22-build
set version 2019.1
set sys linux86

# additional env variables 
set emandir /network/eman1.9
set pythondir $topdir/lib

setenv EMAN2_V "2019.1"
setenv HDF5_DISABLE_VERSION_CHECK 1
setenv EMAN2DIR $topdir
setenv EMANDIR $emandir
setenv PYTHONPATH $pythondir

prepend-path PATH $topdir/include
prepend-path PATH $topdir/bin
prepend-path MANPATH $topdir/man
prepend-path LD_LIBRARY_PATH $topdir/lib:/usr/lib
64

module load python-2.7.13

The first test, loading the module as an user, and running e2version.py, tells me what I want to see.

user@computer ~/test $ > module load eman22-2019.1 
user@computer ~/test $ > e2version.py 
EMAN 2.22 final (GITHUB: 2019-01-23 15:14 - commit: 04f6f33 )
Your EMAN2 is running on: Linux-3-XXXXX-YYYYY-ZZZZ
Your Python version is: 2.7.14

We are done! See you on the other side, soon, I promise…