HOWTO: install cryoDRGN: Deep Reconstructing Generative Networks for cryo-EM heterogeneous reconstruction

Well, don’t be scared because of the picture that I take from this paper. Basically we want to install a python thingy on a conda environment. It should work out of the box. Let me show you what happened in my case. I’m following the github installation procedure.

$ > conda create --name cryodrgn1 python=3.9
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan #
environment location: /XXX/.conda/envs/cryodrgn1
added / updated specs:
- python=3.9
The following packages will be downloaded:
package                    |            build
---------------------------|-----------------
ca-certificates-2023.01.10 |       h06a4308_0         120 KB
libffi-3.4.2               |       h6a678d5_6         136 KB
ncurses-6.4                |       h6a678d5_0         914 KB
openssl-1.1.1t             |       h7f8727e_0         3.7 MB
python-3.9.16              |       h7a1cb2a_0        25.0 MB
wheel-0.38.4               |   py39h06a4308_0          64 KB
-----------------------------------------------------------                                   Total:        30.0 M
The following NEW packages will be INSTALLED:
_libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main 
_openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu 
... some more packages here...
zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0 

Proceed ([y]/n)? y

After this we activate the environment and start customising it.

(cryodrgn1) user@server ~ $ > conda install pytorch -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /XXX/.conda/envs/cryodrgn1
added / updated specs:
- pytorch
The following packages will be downloaded:
package                    |            build
---------------------------|-----------------
blas-1.0                   |              mkl           6 KB
flit-core-3.6.0            |     pyhd3eb1b0_0          42 KB
intel-openmp-2022.1.0      |    h9e868ea_3769         4.5 MB
mkl-2022.1.0               |     hc2b9512_224       129.7 MB
pytorch-1.13.1             |      py3.9_cpu_0        81.6 MB  pytorch-mutex-1.0          |              cpu           3 KB  typing_extensions-4.4.0    |   py39h06a4308_0          46 KB
------------------------------------------------------------                                    Total:       215.9 MB
The following NEW packages will be INSTALLED:
blas               pkgs/main/linux-64::blas-1.0-mkl 
... some more packages here ...
typing_extensions  pkgs/main/linux-64::
typing_extensions-4.4.0-py39h06a4308_0 

Proceed ([y]/n)? y

 Now we install two classics, pandas and scikit. 

(cryodrgn1) user@server ~ $ > conda install pandas
(cryodrgn1) user@server ~ $ > conda install seaborn scikit-learn
(cryodrgn1) user@server ~ $ > conda install umap-learn 
jupyterlab ipywidgets 
cufflinks-py "nodejs>=15.12.0" -c conda-forge

And after this, a whole lotta jupyter lab extensions.

(cryodrgn1) user@server ~ $ > 
jupyter labextension install 
@jupyter-widgets/jupyterlab-manager --no-build
(cryodrgn1) user@server ~ $ > 
jupyter labextension install 
jupyterlab-plotly --no-build
(cryodrgn1) user@server ~ $ >
jupyter labextension install 
plotlywidget --no-build
(cryodrgn1) user@server ~ $ > jupyter lab build 
[LabBuildApp] JupyterLab 3.5.0 
[LabBuildApp] Building in /XXX/.conda/envs/cryodrgn1/share/jupyter/lab 
[LabBuildApp] Building jupyterlab assets (production, minimized)

The pip installation goes very smoothly, and at the end, we are able to run our test command.

git clone https://github.com/zhonge/cryodrgn.git
cd cryodrgn
pip install .

Which is not meaning, of course, that we are going to use the program in a satisfactory way. But a smooth install is a good beginning 🙂

Happy new year and a gute Rutsch!

The source of the image. A random page, of course.

Yes I’ve not been a good blogger the last month, but it has been a little bit crazy out of the internet, so to say. I don’t think I need to explain to you what’s life, so I’ll simply say that I plan to come back to be a good blogger starting … in a few days. I need to read myself a little to find out if I want to make some changes. I’ll probably do some. Maybe. Anyway. The time has come to say it loud and clear,

Happy new year!

And a good landing in January 2023. I’ll read you, or maybe, you’ll read me. My best wishes 💕❤️

I Taught An AI To Write My Book Reviews | RNN Text Generation with Tensorflow

Well, I reblog because this merges also two of my hobbies, so to say: computing and reading…

Frappes and Fiction

A couple weeks ago, I went to my first hackathon with my friend. (A hackathon is a competition where you have to try to code a finished product within a strict time period, so, like, if coding for 24 hours straight is your idea of fun….)

Anyway, we kind of failed– never finished our project, but we DID go to a bunch of the workshops they offered there. One of them was on machine learning, and after attending it, I decided to try to learn more on my own.

For this, I was using Google Colaboratory, an environment that lets you do machine learning and data analysis stuff on the cloud, and an open-source Python library called Tensorflow, which is used for machine learning.

Tensorflow’s website includes a bunch of tutorial code you can use, so I started from their tutorial code for an RNN text generator.

What is an…

View original post 958 more words

High Performance Computing is at Inflection Point

A fascinating approach. Reblogged for my reference and in case you don’t follow already the Linux Cluster blog. 😉

The Linux Cluster

A group of researchers led by Martin Schulz of the Leibniz Supercomputing Center (Munich) presented a paper in which they argue HPC architectural landscape of High-Performance Computing (HPC) is undergoing a seismic shift.

The Full Article is taken from Summer Reading: “High-Performance Computing Is at an Inflection Point”

4 Guiding Principles

• Energy consumption is no longer merely a cost factor but also a hard feasibility constraint for facilities.
• Specialization is key to further increase performance despite stagnating frequencies and within limited energy bands.
• A significant portion of the energy budget is spent moving data and future architectures must be designed to minimize such data movements.
• Large-scale computing centers must provide optimal computing resources for increasingly differentiated workloads.

Ideas Snippets – Integrated Heterogeneity

Integrated Heterogenous Systems are a promising alternative, which integrate multiple specialized architectures on a single node while keeping the overall system architecture a homogeneous collection of mostly identical…

View original post 340 more words

Deploy and use of a Kubernetes cluster on CentOS 7.X (part 3)

Are you having a kubernetes cluster (eine Kubernetesbauernhof 😉😉) already? If not, please go back to my previous posts, Deployment Part 1 and Deployment Part 2. Here we are going to try to use it. But before we collect some useful commands and the docker equivalents. I adapt this data from the official documentation.

Command	Kubernetes	docker
run	kubectl create deployment –image=nginx nginx-app	docker run -d –restart=always -e DOMAIN=cluster –name nginx-app -p 80:80 nginx
list	kubectl get pods -o wide kubeclt get nodes	docker ps -a
attach	kubectl attach -it nginx-app	docker attach container-id
exec	kubectl exec -ti nginx-app — /bin/sh	docker exec -ti container-id /bin/sh
logs	kubectl logs -f nginx-app	docker logs -f container-id
stop		docker stop container-id
remove	kubectl delete deployment nginx-app kubectl delete pod nginx-app	docker rm container-id
version	kubectl version	docker version
info	kubectl cluster-info	docker info

The cheating table

As you see, there’s a correspondence. After all, we want to run a kubernetes cluster to manage our dockers, right? Is it the same or not? Not really, a cluster is there to manage common resources, and here you have the additional functionalities for kubernetes that I’d like to show you. Let’s say you want deploy your nginx using a yaml (yet another markup language) file. This is a standard set of operations you may want to run on your kubernetes master. Operations taken from this post. Also you can use the yaml there, if you don’t have one already.

Create your first deployment:

kubectl create -f nginx.yaml --record

List your deployments:

kubectl get deployments

Check that your Pod is present:

kubectl get pods

Increase the number of replicas to 8:

kubectl scale deployment nginx-server --replicas=8

Decrease the number of replicas to 3

kubectl scale deployment nginx-server --replicas=3

The next one is mine. Describe one pod (that is, learn about what’s going on with it)

## > kubectl describe pod cuda-vector-add
Name:        cuda-vector-add
Namespace:   default
Priority:    0
Node:        ONE/IP-OF-ONE
Start Time:  DATE
Labels:      <none>
Annotations: <none>
Status:      Pending
IP: 
IPs: <none>
Containers:
  cuda-vector-add:
    Container ID: 
    Image: k8s.gcr.io/cuda-vector-add:v0.1
    Image ID: 
    Port: <none>
    Host Port: <none>
    Command:
    /bin/sleep
    3650d
State: Waiting
   Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
   nvidia.com/gpu: 1
Requests:
   nvidia.com/gpu: 1
Environment: <none>
Mounts:
   /var/run/secrets/kubernetes.io/serviceaccount 
   from kube-api-access-bz6sz (ro)
Conditions:
   Type            Status
   Initialized     True 
   Ready           False 
   ContainersReady False 
   PodScheduled    True 
Volumes:
  kube-api-access-bz6sz:
   Type: Projected 
     (a volume that contains injected data from multiple sources)
   TokenExpirationSeconds: 3607
   ConfigMapName:          kube-root-ca.crt
   ConfigMapOptional:     <nil>
   DownwardAPI:           true
QoS Class:      BestEffort
Node-Selectors: accelerator=geForce-gtx-1080
Tolerations:    node.kubernetes.io/not-ready:NoExecute 
                op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
    Type   Reason  Age  From    Message
    ----   ------  ---- ----    -------
    Normal Pulling 69s  kubelet Pulling image 
                               "k8s.gcr.io/cuda-vector-add:v0.1"

I used the cuda-vector-add pod output to illustrate my next kubernetes challenge: to write a yaml for a pod with GPU access and VNC access. I did the very same before with a docker, so I assume it should be possible for a kubernetes pod. But before going for it, a couple of things more on kubernetes cluster configuration. If you don’t have a uniform hardware, like in my case, it is important to label the nodes, so that the pod goes to the right hardware instead of staying on Pending state forever. Like this:

## > kubectl get pods
NAME                  READY STATUS    RESTARTS AGE
cuda-vector-add        0/1  Pending   0        54s
nginx-6799fc88d8-rtsrq 1/1  Running   0        47h
nginx-6799fc88d8-sjjn7 1/1  Running   0        47h

We will label our one GPU node GPU1, rename the label, and unlabel it afterwards. Because, of course, nobody’s perfect! 😁😁. This is done like this:

## > kubectl label nodes GPU1 accelerator=GeForce-GTX-1080
node/sbexp01 labeled
## > kubectl label nodes GPU1 accelerator=geForce-gtx-1080
error: 'accelerator' already has a value 
(GeForce-GTX-1080), and --overwrite is false
## > kubectl label nodes GPU1 accelerator=geForce-gtx-1080 
--overwrite
node/GPU1 labeled
## > kubectl label pod GPU1 accelerator-
Error from server (NotFound): pods "sbexp02" not found
 ## > kubectl label node GPU1 accelerator-
node/GPU1 labeled
## > kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
--> nice table with all node information

And I think that will be it for today. You have quite some material to play. But if not, here you have some more references: How to delete a kubernetes label, How to assign Pods to Nodes, where the nodeSelector is explained, and why my pod is in a pending state. In my case, it was downloading an image. Here the Stackoverflow post about pods on pending state. And here the official debug pod documentation. Tomorrow I may retake the pod setup, if I have time. Stay tuned. Or not, just come back later for more 😉.

W.10 preventing popup asking to overwrite on powershell

Something like this. Image taken from this post.

I have this borrowed PS script to make an MS excel file from a CSV file. A pretty common stuff I guess. It’s not mine so I don’t want to mess with it. The script works fine on my W.10 machine, and I’ve scheduled it, but on the host I get each time after the run an annoying popup window that doesn’t seem to correspond to the reality, since it tells us

A file named 'Results.xlsx' already exists in this location. Do you want to replace it?

but navigating to the folder in question doesn’t show any Results.xlsx. I don’t want, as usual, to blame anyone specifically (hello script writer, hello MS PS, hello Office developer) but to fix it. The solution seems to be to add just one line before the SaveAs statement. Like this:

$excel.DisplayAlerts = $false;
$excel.ActiveWorkbook.SaveAs($xlsFile);

Solution found on this StackOverflow post. Another reference to the issue is on this other post.

Blueutil : bluetooth control through the command line on Big Sur

I bought Philips Hue lights. They are Bluetooth and since they are supposed to be smart I’ve decided to (try to) take control over them. First thing will be to know how Bluetooth works. Basically, you have a network similar to the Wireless network, but hooking on other interfaces (the bluetooth interfaces). As for WiFi, you may have equivalent commands to scan the bluetooth network. For example, the equivalent to ifconfig is hciconfig. The problem is that those tools are not standard but hacking tools, so to get all the juice I’ll need to install Kali on a VM or physical device. Of course I could do that, but I don’t want anything so complicated. I want to run it on my MacBook Pro. Google hcitool for mac brought me here. Unfortunately the repository explanation is not enough and I don’t want to build some unknown swift files. Next entry on my search says directly that hcitools is not available for mac. But it doesn’t need to be hcitools.

First let’s learn to use the native Big Sur bluetooth tools. On this article from apple insider I learn that one can get some kind of “extended” bluetooth panel by simply pressing Shift + Option before clicking on the Bluetooth icon. I took the bluetooth logo above from them also! By the way, it looks like bluetooth has some issues on macOS Big sur, here you have how to fix your bluetooth if you are one of the unlucky people suffering from them.

But let’s get to the meat. This entry on StackExchange told me about blueutil. I have homebrew, so to install it is quite easy. It looks like this:

$ > brew install blueutil
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
... more brew stuff here...
==> Downloading 
https://homebrew.bintray.com/bottles/blueutil-2.7.0.big_sur.bottle.tar.gz
######################################################################## 100.0%
==> Pouring blueutil-2.7.0.big_sur.bottle.tar.gz
🍺  /usr/local/Cellar/blueutil/2.7.0: 6 files, 261KB

That seems to be it. If you don’t have homebrew, here you have instructions about how to install blueutil including brew. And here the git page of blueutil. Internet has it all man! Does it work?

$ > blueutil -p
0

It means I have the bluetooth off. I can switch it on like this

 $ > blueutil -p 1

And now I can list the paired devices, for example.

$ > blueutil --paired
address: MAC-ADDRESS-ONE, 
not connected, not favourite, not paired, 
name: "-", recent access date: -
address: MAC-ADDRESS-TWO, 
not connected, not favourite, paired, 
name: "My-Bluetooth-Thing", 
recent access date: YYYY-MM-DD HH:MM:SS 

So very nice! I managed to get what I wanted. Or didn’t I? I have the MAC addresses of my Philips bulbs so I know that MAC-ADDRESS-ONE is one of my bulbs. What can I do with this? I have no idea yet 😀

CentOS 7.7 HOWTO add multiple KDE desktops

You may need more than one desktop on KDE. By default, at least in my case, only one is present, so let’s add some more. Go to System Settings (lower left corner) ->Workspace behaviour (the yellow flame on the image above) ->Virtual Desktops. You can configure the number of desktops there, the names, and the shortcuts to access to them. Solution found here.

I must admit I’m being a little bit lazy here. I should have written something more dense, but as you know, this is my blog but also my log 😉

shell message another user on CentOS 7.X

This is a short one. I wanted to know if the trick was still possible, since it’s an old one. Let’s say I want to send a message to user “alphonse” that tends to work on my machine. I am connected through ssh. First I check who’s logged in with who.

manuel@machine ~ ## > who
alphonse :0           May  6 16:54 (:0)
alphonse pts/0        May  6 16:55 (:0)
alphonse pts/1        May  6 17:31 (:0)
manuel pts/2        May 13 16:06 (:0)
manuel pts/3        May  8 16:14 (:0)

Now I send a message to his shell:

manuel@machine ~ ## > echo "GET OUT OF HERE!" | write alphonse pts/1

And alphonse will get the message on the shell he’s running on machine. Not very useful, but you never know when it will become handy 😀

docker of the day : netdata

I have a new docker project. It’s top secret, so I can’t tell you about, but it’s going to be really cool. In the meantime, I’m checking the news on the docker community.  Amongst them I found netdata. The above GIF (taken from that very page) is showing you how it really looks like once it’s up an running, that is, like most of the dockers, almost right out of the box. But what is netdata? Netdata is distributed, real-time performance and health monitoring for systems and applications (I quote). What is the plan? The plan is, in principle, to run one netdata docker per “physical” client, and orchestrate them all through kubernetes. That way, we will always have a netdata docker running as service, allowing me to check in real time the client working load through its IP. As easy as

http://server.ip:19999/

The reality after all the hype was a little bit disappointing. I have an already working solution, that is munin, and I know how to write munin plugins so that the show me what I want. In addition, the munin server is under my control, not like netdata cloud. So why netdata? I don’t have a clear answer to that, but hey, I just wanted to write about it. But the difference is there. On munin, we need to have a service running per munin client. The service sends its reports to the munin server (docker or not). On netdata, we have dockers running instead. And they can be under kubernetes control. I think. I don’t know. Or I don’t know yet 🙂