CryoSPARC 2 slurm cluster worker update error

This is about CryoSPARC again. Previously we did install it on CentOS and update it, but on a master + node configuration, not on a cluster configuration. If it’s a new install on your slurm cluster, you should follow the master  installation guide, that tells you to make a master install on the login node, then, on the same login node install the worker:

module load cuda-XX
cd cryosparc2_worker
./install.sh --license $LICENSE_ID --cudapath 

The situation is that we update the master node but the Lane default (cluster) doesn’t get the update and the jobs crash because of it. First we uninstall the worker using one of the management tools like this:

cryosparcm cli 'remove_scheduler_target_node("cluster")'

Then we cryosparc stop and we move the old worker software folder

mv cryosparc2_worker cryosparc2_worker_old

and get a new copy of the worker software with curl.

curl -L https://get.cryosparc.com/\
download/worker-latest/$LICENSE_ID \ > cryosparc2_worker.tar.gz

We cryosparc start, and untar, cd, and install. Don’t forget to add your LICENSE_ID and to load the cuda module or be sure you have one cuda by default. This is an edited extract of my worker install:

******* CRYOSPARC SYSTEM: WORKER INSTALLER ***********************

Installation Settings:
License ID :  XXXX
Root Directory : /XXX/Software/Cryosparc/cryosparc2_worker
Standalone Installation : false
Version : v2.5.0

******************************************************************

CUDA check..
Found nvidia-smi at /usr/bin/nvidia-smi

CUDA Path was provided as /XXX/cuda/9.1.85
Checking CUDA installation...
Found nvcc at /XXX/cuda/9.1.85/bin/nvcc
The above cuda installation will be used but can be changed later.

***********************************************************

Setting up hard-coded config.sh environment variables

***********************************************************

Installing all dependencies.

Checking dependencies... 
Dependencies for python have changed - reinstalling...
---------------------------------------------------------
Installing anaconda python...
----------------------------------------------------------
PREFIX=/XXX/Software/Cryosparc/cryosparc2_worker/deps/anaconda
installing: python-2.7.14-h1571d57_29 ...

...anaconda being installed...
installation finished.
---------------------------------------------------------
Done.
anaconda python installation successful.
---------------------------------------------------------
Preparing to install all conda packages...
-----------------------------------------------------------
----------------------------------------------------------
Done.
conda packages installation successful.
------------------------------------------------------
Preparing to install all pip packages...
----------------------------------------------------------
Processing ./XXX/pip_packages/Flask-JSONRPC-0.3.1.tar.gz

Running setup.py install for pluggy ... done
Successfully installed Flask-JSONRPC-0.3.1 
Flask-PyMongo-0.5.1 libtiff-0.4.2 pluggy-0.6.0 
pycuda-2018.1.1 scikit-cuda-0.5.2
You are using pip version 9.0.1, 
however version 19.1.1 is available.
You should consider upgrading via the
 'pip install --upgrade pip' command.
-------------------------------------------------------
Done.
pip packages installation successful.
-------------------------------------------------------
Main dependency installation completed. Continuing...
-------------------------------------------------------
Completed.
Currently checking hash for ctffind
Dependencies for ctffind have changed - reinstalling...
--------------------------------------------------------
ctffind 4.1.10 installation successful.
--------------------------------------------------------
Completed.
Currently checking hash for gctf
Dependencies for gctf have changed - reinstalling...
-------------------------------------------------------
Gctf v1.06 installation successful.
-----------------------------------------------------------
Completed.
Completed dependency check.

******* CRYOSPARC WORKER INSTALLATION COMPLETE *****************

In order to run processing jobs, you will need to connect this
worker to a cryoSPARC master.

****************************************************************

We are adding a worker that was somehow previously there, so I don’t do anything else. If I check the web, Lane default (cluster) is back. Extra tip: the forum entries about a wrong default cluster_script.sh, and about the Slurm settings for cryosparc v2.

If I need to add something: be aware that the worker install looks like coming with its own python, and it does reinstall cftfind and gctf. So be careful if you run python things in addition to cryosparc 🙂

Advertisements

Install EMAN2.2 with cmake 3.13 on CentOS 7

I’m not going to bloat the post with personal opinions about how good is the installation method is. Here you get the EMAN2 sources. This is the HOWTO install , and from there I did the compilation with anaconda. As you may know from previous posts, we use modules. First we load our python module and upgrade conda like suggested on the error message, if you get it. Then we go ahead with the instructions. I do that on a new window.

module load python-2.7.13
conda install eman-deps=13 -c cryoem -c defaults -c conda-forge

In my case, 27 new packages are installed, 14 packages are updated, 16 packages are downgraded, meaning around 40 packages are downloaded and extracted. Since we do that over the python modul, (I hope) I do need to do this only once. It takes like 10 minutes to be done, but multiply that by 30!  Now I cmake what I got from github. As expected, I get a cmake version error.

eman22-build ## > 
cmake ../eman22-src/ -DENABLE_OPTIMIZE_MACHINE=ON
CMake Error at CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
CMake 3.9 or higher is required. You are running version 3.8.2

-- Configuring incomplete, errors occurred!

Let’s donwload then a new cmake and make a module for it. I get the binary cmake for my linux, the cmake-3.13.3-Linux-x86_64.tar.gz, so I don’t need to compile it. The unzipped folder I place on /network/cmake-3.13.3/. My module looks like this then:

## modules cmake-3.13.3
## modulefiles/cmake-3.13.3. Sample gcc module
proc ModulesHelp { } {
global version modroot
   puts stderr "cmake-3.13.3"
}
module-whatis "Sets the environment for using cmake-3.13.3"

# for Tcl script use only
set topdir /network/cmake-3.13.3
set version 3.13.3
set sys linux86

setenv CMAKE_V "3.13.3"

prepend-path PATH $topdir/bin
prepend-path MANPATH $topdir/man
prepend-path LD_LIBRARY_PATH $topdir/lib

Time to cmake again. I do it on a new terminal. First I load both modules, my python and my cmake. Then I just follow the instructions…

cmake ../eman22-src/ -DENABLE_OPTIMIZE_MACHINE=ON

And make -j 4, make install. OK! The final module (eman22 installed on /network/eman22-build) looks like this:

#################
## modules eman22-2019.1
## modulefiles/eman22-2019.1 EMAN module
##
proc ModulesHelp { } {
global version modroot
   puts stderr "eman22-2019.1"
}
module-whatis "EMAN 22 2019.1"
# for Tcl script use only
set topdir /network/eman22-build
set version 2019.1
set sys linux86

# additional env variables 
set emandir /network/eman1.9
set pythondir $topdir/lib

setenv EMAN2_V "2019.1"
setenv HDF5_DISABLE_VERSION_CHECK 1
setenv EMAN2DIR $topdir
setenv EMANDIR $emandir
setenv PYTHONPATH $pythondir

prepend-path PATH $topdir/include
prepend-path PATH $topdir/bin
prepend-path MANPATH $topdir/man
prepend-path LD_LIBRARY_PATH $topdir/lib:/usr/lib
64

module load python-2.7.13

The first test, loading the module as an user, and running e2version.py, tells me what I want to see.

user@computer ~/test $ > module load eman22-2019.1 
user@computer ~/test $ > e2version.py 
EMAN 2.22 final (GITHUB: 2019-01-23 15:14 - commit: 04f6f33 )
Your EMAN2 is running on: Linux-3-XXXXX-YYYYY-ZZZZ
Your Python version is: 2.7.14

We are done! See you on the other side, soon, I promise…

Scipion 1.2 cannot find -lmpi_cxx on CentOS 7

I did install previously a Scipion 1.1 module on CentOS 7 that was linking scipion all the existing programs, including xmipp. Now I want to install the newest version with the default programs, instead of linking it to the existing ones. I followed the How to Install Scipion from sources, with our particularities. One, we use a network folder. Two, we use modules. I will, anyhow, minimize the last point, and I will load only mpi-3.0.0.

So we get the git (I like how it sounds, yeah!) and provided we have the dependencies installed and we have configured the scipion.conf file we are ready to compile it. We want to use the defaults, that is:

./scipion install -j 5

or something similar. Unfortunately, after quite some (compilation) time the install fails with this error:

mpi-3.0.0/lib -lXmippAlglib -lXmippBilib \
-lXmippCondor -lXmippDelaunay -lXmippData \
-lXmippRecons -lXmippClassif -lsqlite3 \
-lfftw3 -lfftw3_threads -ltiff -ljpeg -lpng \
-lhdf5 -lhdf5_cpp -lmpi -lmpi_cxx -lXmippParallel 
/usr/bin/ld: cannot find -lmpi_cxx
collect2: error: ld returned 1 exit status
scons: *** [software/em/xmipp/bin/
xmipp_mpi_angular_continuous_assign] Error 1
scons: building terminated because of errors.

What now? Google doesn’t have an answer to the lmpi_cxx problem, but don’t cry yet. What if we comment out the request for that library?

root # > grep 'mpi_cxx' software/em/xmipp/*
...some more stuff here...
grep: software/em/xmipp/resources: Is a directory
software/em/xmipp/scipion_sconscript: 
kwargs['libs'] += ['mpi', 'mpi_cxx', 'XmippParallel']

Let’s edit the file scipion_sconscript with the request and try again.  On the line 464, instead of

kwargs['libs'] += ['mpi', 'mpi_cxx', 'XmippParallel']

we write

kwargs['libs'] += ['mpi', 'XmippParallel']

We save the file, and run

./scipion install -j 5

again. This time, everything goes fine 🙂

How about the post-installation tests? In my case, everything went fine…provided you run as root. Or some tests will fail due to a weird Permission Denied error.

>>>> python scripts/sync_data.py –download xmipp_tutorial
Selected datasets: xmipp_tutorial
Dataset xmipp_tutorial not in local machine. Downloading…
Traceback (most recent call last):
File “scripts/sync_data.py”, line 440, in <module>
main()
File “scripts/sync_data.py”, line 128, in main
download(dataset, url=args.url, verbose=args.verbose)
File “scripts/sync_data.py”, line 257, in download
os.makedirs(datasetFolder)
File “/usr/local/scipion/software/lib/python2.7/os.py”, line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: ‘/usr/local/scipion/data/tests/xmipp_tutorial’

Yes, what can I say? Be root, my friend. I’m not going to write down the scipion-1.2 module, I hope you’re fine with that already. Or if not, just copy the one of scipion-1.1, remove the python module call and change the path so that it points out to the new install 🙂

 

Install pyEM on CentOS 7

After a week of silence, I’m back on track. I managed to integrate pyEM onto our systems, and this is my log about. We want it to export cryoSPARC data to relion. Since we don’t start from scratch, my first trial is over one of the python modules, so that I do it once and I have it everywhere. This is not the preferred method. And it didn’t work in our case. Let me show you how to get to the error.

module load python-2.7.13
git clone https://github.com/asarnow/pyem.git
cd pyem
pip install -e .

The install goes on for a while, until this:

Downloading stuff ...  
Matplotlib 3.0+ does not support Python 
2.x, 3.0, 3.1, 3.2, 3.3, or 3.4.
Beginning with Matplotlib 3.0, Python 3.5 and above is required.
This may be due to an out of date pip.
Make sure you have pip >= 9.0.1.
----------------------------------------
Command "python setup.py egg_info" failed with 
error code 1 in /tmp/pip-build-H4XfvY/matplotlib/
You are using pip version 8.1.2, however 
version 18.1 is available.
You should consider upgrading via the 
'pip install --upgrade pip' command.

Of course it can be pip. Before upgrading it, I try with another python module, that is:

module load python-3.5.3
git clone https://github.com/asarnow/pyem.git
cd pyem
pip install -e .

The error in this case, is with the default gcc.

/bin/ld: cannot find -lpython3.5m
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pyfftw

I have tried yum install python-devel gcc-g++, but it doesn’t fix it. Another trial is yum install python34-devel. But still gcc fails. I make the python-3.5.3 module local by copying it to the local drive and this time I manage. Alright, we know it works. It’s not very convenient having it locally only, so I go for miniconda. Step by step install is here.

Step 1: Install miniconda

 bash Miniconda2-latest-Linux-x86_64.sh -u

I do it on a network location (NFS mounted) rewriting an existing miniconda install.

Step 2: On a new shell, export the path to miniconda, git clone the folder onto a network location (the same NFS mounted share)

git clone https://github.com/asarnow/pyem.git

and pip install it.

Step 3: Write a module for it. In my case, the network drive is /network/. The module looks like this:

#%Module1.0#####################################
## modules pyem
## modulefiles/pyem pyEM
##
proc ModulesHelp { } {
global version modroot
    puts stderr "pyem - sets the Environment for using pyem"
}

module-whatis "Sets the environment for using pyem-0.3"

# for Tcl script use only
set topdir /network/pyem-0.3
set version 0.3
set sys linux86
prepend-path PATH /network/miniconda2/bin:$topdir:$topdir/pyem/

And that’s what I have to say today. I hope it helps somebody!

 

Topaz install on CentOS 7

Yes I’m back to work. I should have logged it more, but I was busy with so many issues that the only thing I wanted to write was fiction. Anyway. Today we are going to install Topaz over our already existing python module.

You may wonder what topaz is: it  is a “pipeline for particle detection in cryo-electron microscopy images using convolutional neural networks trained from positive and unlabeled examples“. Understood? Alright, so we can go ahead. On the github topaz page you find  the software installation formula and a docker definition. I did try the docker, but for some reason linked with my default python, my docker was created but I couldn’t manage to make it run.  I call these ones DOA dockers (Dead On Arrival).

When I find a DOA docker I tend to reverse engineer it. In this case, it is not needed. We have a “From source” section.  I quote:

Installation of dependencies

conda install numpy pandas scikit-learn cython
conda install -c soumith pytorch=0.2.0 torchvision
conda install -c soumith pytorch torchvision cuda80

This takes for a while. In my case, around 20 minutes, but I may be influenced by the installation method. Then I cd into my topaz folder (the one of the DOA docker) and type

pip install .

No error is thrown. To test it’s properly installed, I go to another machine, load my python module and try to do “topaz” something. And it works! The users are going to be happy, since my python module is mounted on a network location, everybody will have topaz once they load it. Even better than a docker! We’ll see what they can do with it.

CCP-EM install on CentOS 7

The CCP-EM software suite is a package containing tools for cryo-EM. You can find it here. I want to install it together with the already existing software. The Brief instructions seem to work fine, provided you have the requirements. This is just a reminder of the errors you get if you don’t have them. If you don’t have ccp4:

./install_ccpem.sh

################################ Install CCP-EM 
Setting up paths...
Compiling py-files

Warning: CCP4 not found!

Several CCP-EM tasks use CCP4 programs. 
To use those tasks, please install
and set up CCP4 before running CCP-EM. 
CCP4 is available from www.ccp4.ac.uk.

I had a module for ccp4, so I can load it and continue. If you don’t have modeller:

openf______E> Cannot open file 
/something/modeller-9.18/modlib/libs.lib: 
No such file or directory

Traceback (most recent call last):
...
File "/my/python2.7/site-packages/modeller/__init__.py", 
line 99, in <module>
_modeller.mod_start()
IOError: openf______E> Cannot open file 
/something/modeller-9.18/modlib/libs.lib: 
No such file or directory

Simpy install modeller and reinstall ccpem. Yah, lame post, only with errors. But sometimes you feel better knowing other people commit the same mistakes you commit. I will write other things as soon as my muse is back 😛

 

crYOLO conda error: bad interpreter

Creating a new virtual environment for crYOLO, that implement You Only Look Once for particle picking, I stumbled upon this error. I saw it before, but here we don’t use python so much, I mean, we use it only for installations. Now I have two versions of python compiled and working with their corresponding modules. The reason to have this is to be independent of the Linux client (and the updates it may get by mistake). I already installed the last Sphire version and wrote the corresponding module. So the error is:

module load python-2.7.13
conda create -n cryolo python=2 pyqt=5
-bash: /my/anaconda2/bin/conda: 
/usr/local/anaconda2/bin/python: bad interpreter: 
No such file or directory

I’m probably guilty here. It seems that the installed conda looks for the “native” python, that is installed on /usr/local/anaconda2/bin/. So how do we fix it? As usual, somebody on StackOverflow already found this error before. I first localize my python.

which python
/my/anaconda2/bin/python

Then I replace, as suggested, the shennanigan on my conda.

which conda
/my/anaconda2/bin/conda
vi /my/anaconda2/bin/conda

On conda, where it is written /usr/local/anaconda2/bin/python I write /my/anaconda2/bin/python. After this small modification, I’m able to create the cryolo virtual environment and so on. I love simple solutions 🙂