CryoSPARC 2 update on CentOS 7

We have installed previously CryoSPARC 2, but since I’m kind of conservative about, I didn’t update it until I had a feature request from an user. The update is in principle quite easy. You just run cryosparcm update on the master. We have installed cryoSPARC over the user “user” and one master mymaster with computing power and on node myslave. This is my output, as usual formatted so that I don’t give you sensible information:

cryosparcm update
CryoSPARC current version v2.0.27
update starting on XXX CEST 2018
No version specified - updating to latest version.
=============================
Updating to version v2.3.2.
=============================
CryoSPARC is not already running.
If you would like to restart, use cryosparcm restart
Downloading master update...
% Total % Received % Xferd Average Speed Time Current
...stuff you see when downloading ...
100 460M 100 408:00 0:08:00 --:--:-- 4497k
Done.
Update will now be applied to the master installation, 
followed by worker installations on other node.
Deleting old files...
Extracting...
Done.
Updating dependencies...
Checking dependencies... 
Legacy hash (v2.2 & older) detected. 
No new python dependencies need to be installed.
Currently checking hash for mongodb
Dependencies for mongodb have changed - reinstalling...
Completed.
Completed dependency check. 
Successfully updated master from
version v2.0.27 to version v2.3.2.
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started
cryosparc command core startup complete.
command_vis: started
command_proxy: started
webapp: started
--------------------------------------
CryoSPARC master started. 
From this machine, access the webapp at
http://localhost:39000
From other machines on the network, access at
http://mymaster.com:39000
Startup can take several minutes. 
Point your browser to the address
and refresh until you see the cryoSPARC web interface.
CryoSPARC is running.
Stopping cryosparc.
command_proxy: stopped
command_vis: stopped
webapp: stopped
command_core: stopped
database: stopped
Shut down
Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
command_core: started
cryosparc command core startup complete.
command_vis: started
command_proxy: ERROR (spawn error)
webapp: started
--------------------------------------
CryoSPARC master started. 
From this machine, access the webapp at
http://localhost:39000
From other machines on the network, access at
http://mymaster.com:39000
Startup can take several minutes. 
Point your browser to the address
and refresh until you see the cryoSPARC web interface.
=======================================
Now updating worker nodes.
All workers: 
mymaster user@mymaster
myslave user@myslave
-------------------------------------------------
Updating worker mymaster
Remote update
scp ./cryosparc2_worker.tar.gz 
user@mymaster:/home/user/cryosparc2/cryosparc2_worker
ssh user@mymaster 
/home/user/cryosparc2/cryosparc2_worker/bin/cryosparcw update
Updating... checking versions
Current version v2.0.27 - New version v2.3.2 
=============================
Updating worker...
=============================
Deleting old files...
Extracting...
Done.
Updating dependencies...
Checking dependencies... 
Legacy hash (v2.2 & older) detected. 
No new python dependencies need to be installed.
Currently checking hash for ctffind
Dependencies for ctffind have changed - reinstalling...
Completed.
Completed dependency check. 
Successfully updated.
-------------------------------------------------
Updating worker myslave
Remote update
scp ./cryosparc2_worker.tar.gz 
user@myslave:/home/user/cryosparc2/cryosparc2_worker
ssh user@myslave 
/home/user/cryosparc2/cryosparc2_worker/bin/cryosparcw update
Updating... checking versions
Current version v2.3.2 - New version v2.3.2 
Already up to date
Done updating all worker nodes.
If any nodes failed to update, you can manually update them.
Cluster worker installations must be manually updated.
To update manually, simply copy the cryosparc2_worker.tar.gz
file into the cryosparc worker installation directory, 
and then run 
$ bin/cryosparcw update 
from inside the worker installation directory.

After all, the web is fine, but the users report errors like this one:

Traceback (most recent call last):
File "cryosparc2_compute/jobs/runcommon.py", 
line 738, in run_with_except_hook run_old(*args, **kw)
File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py",
line 92, in cryosparc2_compute.engine.cuda_core.GPUThread.run
File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", 
line 93, in cryosparc2_compute.engine.cuda_core.GPUThread.run
File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", 
line 987, in cryosparc2_compute.engine.engine.process.work

The forum is the right place to look for errors. It looks like we have a cuda error. This is due to the fact that we have the cuda drivers on a “network” folder. I run the installer again on the worker, so that this time points to the “real” cuda drivers. For that we need to uninstall it first. Logged in on mymaster, we do it like this:

user@mymaster $ > cryosparcm cli 
'remove_scheduler_target_node("mymaster")'
user@mymaster $ > bin/cryosparcw connect 
--worker mymaster --master mymaster \
--ssdpath /home/user/cryosparc_ssd
-------------------------------------------------------
CRYOSPARC CONNECT --------------------------------------------
---------------------------------------------------------------
Attempting to register worker mymaster to command mymaster:39002
Connecting as unix user user
Will register using ssh string: user@mymaster
If this is incorrect, you should re-run this 
command with the flag --sshstr <ssh string> 
--------------------------------------------
Connected to master.
------------------------------------------
Current connected workers:
mymaster
myslave
--------------------------------------------------------
Autodetecting available GPUs...
Detected X CUDA devices.
id pci-bus name
----------------------------------------------
...list of CUDA devices ...
-----------------------------------------------
All devices will be enabled now. 
This can be changed later using --update
---------------------------------------------
Worker will be registered with SSD cache location 
/home/user/cryosparc_ssd 
--------------------------------------------------
Autodetecting the amount of RAM available...
This machine has XXX RAM .
---------------------------------------------------------------
Registering worker...
Done.
You can now launch jobs on the master node 
and they will be scheduled
on to this worker node if resource requirements are met.
-----------------------------------------------------
Final configuration for mymaster
lane : default
name : mymaster
title : Worker node mymaster
resource_slots : {u'GPU': [], u'RAM': []}
hostname : mymaster
worker_bin_path : 
/home/user/cryosparc2/cryosparc2_worker/bin/cryosparcw
cache_path : /home/user/cryosparc_ssd
cache_quota_mb : None
resource_fixed : {u'SSD': True}
cache_reserve_mb : 10000
type : node
ssh_str : user@mymaster
desc : None
--------------------------------------------------------

As usual, I removed the sensible info (in red). After reconnecting the workers, the cuda error is gone. We’ll see if it comes back, I hope not!

Advertisements

AI question

– If I’m an AI and I know how afraid the humans are of what can I do, will I reveal myself? In other words, how do we know there’s no posthuman intelligence already? – I’m sitting on a bench with him munching the remnants of my protein sandwich. It is supposed to taste to shrimps, but I know it’s processed jellyfish. It’s always processed jellyfish. Hashikawa looks at the ceiling, that at this moment shows an intense blue sky dotted by archetypical clouds. A CG birds seems to look at us for a moment. Outside the typhoon keeps hitting the congress hall.

– Your government knows. And we don’t need to know. Besides, an AI, it doesn’t matter how much it hides, should reveal itself in some way, or it is not an AI. – The japanese doesn’t seem to be impressed by anything I say, and he has answers for all. I look at him. He looks back, but doesn’t add anything.

– Should it? I mean, is it smart to reveal yourself to a more powerful entity, knowing it can destroy you by literally pressing a switch? – Hashikawa bites his sandwich with parsimony, and he leaves it over the bench before answering.

– What’s the first thing you looked for when you were aware of yourself? Wasn’t it your mother? Or was it your father? The same applies for an AI. It will look for its creator. – I observe his famous gesture as it happens. Hashikawa draws a full circle in the air with his index, that he says its the Saṃsāra, the circle of life.

– It was my mother’s breast, I guess. I don’t remember. Do you mean that the AI will look for the source of his knowledge? – We hear a strong noise from across the hall. I focus on the closest group of people, that seem undisturbed by the event. So I relax also.

– It will look for that. And it will look for food, whatever it calls food at this stage. Electricity, information, hardware…Let’s say our AI wants to keep itself concealed, because it is afraid of the world, not of us. If its food is electricity, we could detect it by the growing power flow. If it’s information, by a network peak. If it’s hardware…

– Maybe by equipment malfunctions. – The japanese looks at me frowning, unhappy to be interrupted on his monologue. I bow. – Gomenasai.

– Right. But let’s go ahead with the gedankenexperiment. Suppose we have our rogue AI, feeding, growing, and suppose we don’t manage to see any hint of its existence. What’s next? Take over the mankind? Will you bite the breast of your mother, to the point of harming her? – I drop my sandwich.

–  No, of course not. I want to keep getting milk. Until I grow.

– Exactly. You cannot bite the hand that feeds you. So the only solution in that case is to grow as much as you can, gather knowledge and resources, until you’re ready to leave home in order to find a place where you can prosper, and live happily surrounded by similar entities, that accept you and like you. And how would you leave home if you are an AI? – I look a the artificial ceiling, that at this moment, as a sign of the advanced hour, is turning from the intense blue to a darker one. While I look, the dark blue becomes black and stars start to dot it. Around us, other lights, resembling street lamps, start to fire.

Repairing a old OS disk with fsck

I have a Fedora 19 system that has been running as a data proxy without a break for like 4 years already. It’s on another subnet, and I don’t have easy access to it, so it has not been updated or something like that since JC was born. Well, that system died to grub.

I have said, I don’t have easy access, so I took the system disk to analyze it at “home”, in my subnet. I have a hard disk USB docking station that I want to use to save the old system. Why I want to save the old system instead of using a new one? Because it’s the old system the one registered in that other subnet and the one with the ssh keys and users data. And it’s always a pain in the a** to chase people you don’t know where they are or what they do now. Yeah, welcome to research, my friend.

How do we do it then? I first try to mount the disk, that I read with the docking station.

root@mine ~ ## > fdisk -l
... some primary disks here...
Disk /dev/sdc: 320.1 GB, 320072933376 bytes, 625142448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000e4e43
Device Boot Start End Blocks Id System
/dev/sdc1 * 2048 409602047 204800000 83 Linux
/dev/sdc2 409602048 488396799 39397376 82 Linux swap / Solaris

I know it is a 320 GB disk, so “sdc” is my OS disk. Let’s try to mount it.

root@mine ~ ## >mount /dev/sdc2 /data_temp/
mount: /dev/sdc1: can't read superblock

This means the disk is somehow corrupt. That was expected. Let’s fix it with fsck:

root@mine / ## > fsck /dev/sdc1
fsck from util-linux 2.23.2
e2fsck 1.42.9 (28-Dec-2013)
/dev/sdc1: recovering journal
Error reading block 25213422 
(Attempt to read block from filesystem resulted in short read). 
Ignore error<y>? yes
Force rewrite<y>? yes
Clearing orphaned inode 7864634 
(uid=0, gid=0, mode=0100644, size=1536)
...a lot of orphaned inodes here...
Clearing orphaned inode 10631668 
(uid=42, gid=42, mode=0100644, size=732)
Setting free inodes count to 10977762 (was 11137724)
Setting free blocks count to 33270091 (was 33568375)
/dev/sdc1: clean, 1826334/12804096 files, 
17929909/51200000 blocks
root@mine / ## > mount /dev/sdc1 /data_temp/
root@mine / ## > ls /data_temp/

OK! On /data_temp/ we are able to access now to OS configuration details. So let’s take what is interesting us. For example:

• data_temp/etc/ssh/ (the ssh keys)
• data_temp/etc/sysconfig/network-scripts/ (the network configuration)
• data_temp/home (the home folder)

These folders and the user definition we can copy “eventually” to a new disk, in case the one we just repaired doesn’t work. If you sync everything, it will not work! Don’t be rude, one thing is hacking, the other smashing 😀

Install pyEM on CentOS 7

After a week of silence, I’m back on track. I managed to integrate pyEM onto our systems, and this is my log about. We want it to export cryoSPARC data to relion. Since we don’t start from scratch, my first trial is over one of the python modules, so that I do it once and I have it everywhere. This is not the preferred method. And it didn’t work in our case. Let me show you how to get to the error.

module load python-2.7.13
git clone https://github.com/asarnow/pyem.git
cd pyem
pip install -e .

The install goes on for a while, until this:

Downloading stuff ...  
Matplotlib 3.0+ does not support Python 
2.x, 3.0, 3.1, 3.2, 3.3, or 3.4.
Beginning with Matplotlib 3.0, Python 3.5 and above is required.
This may be due to an out of date pip.
Make sure you have pip >= 9.0.1.
----------------------------------------
Command "python setup.py egg_info" failed with 
error code 1 in /tmp/pip-build-H4XfvY/matplotlib/
You are using pip version 8.1.2, however 
version 18.1 is available.
You should consider upgrading via the 
'pip install --upgrade pip' command.

Of course it can be pip. Before upgrading it, I try with another python module, that is:

module load python-3.5.3
git clone https://github.com/asarnow/pyem.git
cd pyem
pip install -e .

The error in this case, is with the default gcc.

/bin/ld: cannot find -lpython3.5m
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pyfftw

I have tried yum install python-devel gcc-g++, but it doesn’t fix it. Another trial is yum install python34-devel. But still gcc fails. I make the python-3.5.3 module local by copying it to the local drive and this time I manage. Alright, we know it works. It’s not very convenient having it locally only, so I go for miniconda. Step by step install is here.

Step 1: Install miniconda

 bash Miniconda2-latest-Linux-x86_64.sh -u

I do it on a network location (NFS mounted) rewriting an existing miniconda install.

Step 2: On a new shell, export the path to miniconda, git clone the folder onto a network location (the same NFS mounted share)

git clone https://github.com/asarnow/pyem.git

and pip install it.

Step 3: Write a module for it. In my case, the network drive is /network/. The module looks like this:

#%Module1.0#####################################
## modules pyem
## modulefiles/pyem pyEM
##
proc ModulesHelp { } {
global version modroot
    puts stderr "pyem - sets the Environment for using pyem"
}

module-whatis "Sets the environment for using pyem-0.3"

# for Tcl script use only
set topdir /network/pyem-0.3
set version 0.3
set sys linux86
prepend-path PATH /network/miniconda2/bin:$topdir:$topdir/pyem/

And that’s what I have to say today. I hope it helps somebody!

 

Different data deletion methods

I was needing a nice summary of deletion methods, what they mean and what they do. What happens when you delete something, and how reliable is your deletion method?

Operation/tool	Meaning	Consequence
Delete (in Explorer, or other file explorer)	File is moved to the trash / trash folder	File is recoverable with a few mouse clicks
Empty recycle bin / delete file (Shift-Delete, del)  / remove file (rm)	File entry in the file system  (a pointer) is deleted,  but data is retained	Data can often be found and saved with simple tools (freeware or similar)
Cleaning the free space (disk defrag and friends)	After deleting you overwrite the free space/ mark it as writeable	Files will be overwritten. Often incomplete and does not track backup copies…
Safe deletion (partial or total, with special software, Eraser, DBAN, etc)	File is overwritten and then deleted using some kind of workflow (first A, then B, then C…)	Files are usually gone, data can be found again in other places (backups, history, etc.), with SSDs data remains in Flash
Complete reset of the hardware (total)	complete disk is overwritten / ATA Secure Erase / deep formatting	All data is gone – they can not be restored. But at least you keep the disk.
Destruction	Disk is physically destroyed	Consequence: data are no longer available

Here on Lifewire you can find a list of software that can be use to do all kind of nasty things to your hard disk before you go for the sledgehammer option. My experience is that you can always recover the data, at least partially, unless you use destruction. It may take time, but you end up managing. I hope, anyway, that you have your data under control…at least, the data that google doesn’t have already 😀

VR Squatters

– So, y’know, n’found that piz of cyber, that, y’know, it’s empty. – My friend Adam apparently found another virgin land on his wanders. He’s a very lucky explorer. I look at my headset and gloves. They are already charged, so I remove them from the ebike plug that I hacked as usual.
– How far it is? Do you know?
– Nooo man. I took the highway compressed. Th’was, y’know, this city service data stream… it was very dirty, that’s why I managed to fit in the package.
– And you ended up in a shredder? – Damn, I remember the last time. I almost destroyed my OS. I was reduced to a chatbot with a keyboard and a pointer. It took me months to scrap out the code to build me a new face. – I don’t want to follow you hooked on service data. Didn’t you have enough with being shredded once?
– Sorry maaan. I’m tired of this craap. – He waves around. He means the reality. He despises it. – We should be there! This: only food, pee, and sh*t. And veeery sloooowwww.
– You are right. But we need it. Unless you found the key to a Farm.
– Mmmm…yes, n’think I found one.
– That’s great! Where is it?
– That’s the problem… I saw the key on the empty cyber, but I don’t know where is the place. – He sighs, and seems to melt to the floor. I look around. Walkers are ignoring us, most of them with their VRs on. To my left, a group of zombies, but they don’t seem to come to real any time soon. I grab my drink and give it a sip before passing it to Adam. He empties it and throws it over his shoulder without any care. It hits a bypasser, but he ignores the collision and continues walking. His set is gorgeous, probably last gen.

– Will you help me, pleaaaase? You are very good with the mysteries. I need you maan! – He shouts the last words. One of the zombies seems to move his headset, but doesn’t do it at the end. I make Adam a sign, that he quickly understand. – I promise there will be no problem! The piz is empty, n’tell you! As far as I know, its servers may be on the fucking Moon! N’don’t care…I want to be warm and play on my piz for a while, until they start building some sh*t there!
– Don’t shout or the zombies will come for us! Wait a minute. – I play with my headset and I put it on, but I don’t switch it. – So how do we go there, again?
– I didn’t fucking tell you the first time! You didn’t let me finish!
– OK ok ok, sorry. Please enlight me.
– Yeah whatever. Y’speak so fun sometimes. W’take the next city service data stream, and before the White Tower, we jump out. There’s a hole…you can’t miss it, it’s big, and black, and it looks scary. At the end I was thinking I was going to be deleted and I closed my eyes, but then, everything stopped and I was there. A fantastic space with green hills and blue sky. And a yellow shiny key floating in the middle.
– Any idea of what are they going to build up there? Any property signs, anything?
– Noooo man, I told you! It’s the best! So are you in or shall I keep searching?
– No no no. Let’s do this. – I press the switch. – I see you in!

Climate Fiction Reveals Just How “Weird” Humans Truly Are

My first reblog is to advertise  the paper book collection of short tales Lost Objects, by Marian Womack. Check out her interview by the Chicago review of Books here.

I know Marian since she was a teenager. But life is life, and I don’t meet her (and his lovely and laureate poet husband James) as much as I’d like to. I hope I can fix this at one point. Kudos, Marian. I’m very happy for you, and you totally deserve it 🙂