GPFS : force deletion of a node

On the spirit of this being my notebook โค๏ธ I want to write down my experience forcing the deletion of a GPFS node. The disk of the node ‘deadnode‘ died without a previous warning (or was it that I wasn’t looking?) so I didn’t have a chance to properly stop the GPFS and so on. Let’s go and force the deletion

quorum ~ ## > mmdelnode -N deadnode
Verifying GPFS is stopped on all affected nodes ...
The authenticity of host 'deadnode (1.2.3.4)'
can't be established.
Are you sure you want to continue connecting (yes/no)? yes
deadnode: bash: /bin/ksh: No such file or directory
mmdelnode: Unable to confirm that GPFS is stopped
on all of the affected nodes.
Nodes should not be removed from the cluster
if GPFS is still running.
Make sure GPFS is down on all affected
nodes before continuing.
If not, this may cause a cluster outage.
mmdelnode: If the affected nodes are permanently down,
they can be deleted with the --force option.
mmdelnode: Command failed.
Examine previous error messages to determine cause.
quorum ~ ## > mmdelnode -N --force deadnode
mmdelnode: Incorrect extra argument: deadnode
Usage:
mmdelnode {-a | -N {Node[,Node...] | NodeFile | NodeClass}}
quorum ~ ## > mmdelnode -N deadnode --force
Verifying GPFS is stopped on all affected nodes ...
deadnode: bash: /bin/ksh: No such file or directory
mmdelnode: Unable to confirm that GPFS is stopped
on all of the affected nodes.
Nodes should not be removed from the cluster
if GPFS is still running.
Make sure GPFS is down on all affected nodes before continuing.
If not, this may cause a cluster outage.
Do you want to continue? (yes/no) yes
mmdelnode: Command successfully completed
mmdelnode: Propagating the cluster configuration data
to all affected nodes.
This is an asynchronous process.

If you don’t know what’s a GPFS filesystem, just search on this very blog. There’s plenty of information above. Maybe too much information. Maybe too much. ๐Ÿ˜๐Ÿ˜๐Ÿ˜

kernel:NMI watchdog: BUG: soft lockup on CentOS 7.X

I saw the soft lockup bug before, but I didn’t write the fix. On my AMD servers, sometimes I see printed on the shell a lot of the following messages:

May 25 07:23:59 XXXXXXX kernel: [13445315.881356] BUG: soft lockup – CPU#16 stuck for 23s! [yyyyyyy:81602]

The date, the CPU and the duration it’s stuck varies. Your analysis will still run, the machine is still usable, but the messages can become so annonying (specially if you have a lot of CPUs) that may not let you work. I found this post for SUSE that is really similar to what happens in CentOS, so I gave it a try. First I check that the watchdog_thresh is defined

hostname:~ # more /proc/sys/kernel/watchdog_thresh
hostname:~ # 10

Then I do as suggested:

hostname:~ # echo 20 > /proc/sys/kernel/watchdog_thresh

And so far the message is not coming back. Problem solved! ๐Ÿ˜Š๐Ÿ˜Š

How to use HP_RECOVERY partition on windows 7

Things get broken, things get tainted… Windows 7 is no more under support but I still have some PCs running it. And stupid me one of them didn’t have a backup copy of the initial working system, since it was installed in 2014. Also I found the original CD but it was… let’s say… unreadable. Actually, the metallic cover of the CD peeled off.

Time to recover it. First trial is to install a new W.7 system on a new disk, and use the activation code on the sticker to activate it. Unfortunately the OA (Online Activated) windows can’t be re-installed, so I ended up with a brand new, unregistered, Windows 7. I tried with different Windows 7 dumped versions and with a Windows 10, with partial success. But I don’t want to write about that. The thing is I was trying to solve the problem the wrong way. With a working OS I can read the “damaged” system disk and I see that it comes with a partition called HP_recovery. Why should I know? I never used the computer! They never called me to check it before! No one cared until now! ๐Ÿ˜”๐Ÿ˜”๐Ÿ˜”

Anyway, the super-expensive hardware is connected to a HP desktop, so the best is to use the HP system recovery for Windows 7. Basically, since I can’t boot, neither recover my install from a recovery point, I’m forced to perform a full HP recovery. This takes time and it’s destructive, so first I clone the original disk, in case I need to extract some license file from it, then I boot the computer pressing F11 repeatedly. At one point, an HP splash screen appears and I see some shell scripts running, then I need to wait (maybe you don’t) until a new window pops up. Sorry guys, no screenshot, but there are not so many options on the new window: repair by wiping up everything or cancel. I choose the first, click on I accept all the risks (nice) and wait for a few hours until I magically have a W.7 back. This time, with license. Now I can tell the user to install his programs to control the hardware, or contact the company that supplies it. And I wonder….for how long this agony called W.7 is going to be with us? ๐Ÿ˜”๐Ÿ˜”๐Ÿ˜”

ERROR: MediaWiki not showing up after a reboot, skin missing

The missing skin message

I like the motto If something works don’t touch it. Unfortunately in the era of CI/CD we need to touch it, and sometimes we even need to reboot it. This is what happened to me. There’s this nice MediaWiki 1.31 install that has been running since years in my team not as a docker or a fancy VM but as a real web server, with its httpd and mariadb services. I run weekly backups but I don’t update it precisely because I was afraid of it. For a reason. After a stupid power cut, when I managed to have the web server back online, I found out that the httpd and mariadb service were running but the MediaWiki was showing a Blank Page. As written on the link before, I go to my /var/www/html/ to edit the LocalSettings.php. Right at the top I copy:

error_reporting( E_ALL );
ini_set( 'display_errors', 1 );

and do systemctl restart httpd. The error instead of the Blank Page reads like this:

Fatal error: Uncaught Exception: /var/www/html/skins/MonoBook/skin.json 
does not exist! in /var/www/html/

and similar. I do check and for some weird reason (update? the other sysadmin?) the skin folder is gone. I then try to disable the skins by modifying the LocalSettings.php and restating the httpd. It works somehow. I get the wiki content unskinned, with an error on the top like this:

The wrong skin message

What to do now? I went for downloading the whole wiki zip again (the same version), copying the skins folder, uncommenting the LocalSettings.php and restating the httpd. And the wiki is back. What to do next? An invisible migration to a more reliable setup, maybe a kubernetes pod. We’ll see if I have time. Or mood. Or actually, both ๐Ÿ˜‰.

GPFS : cannot delete file system / Failed to read a file system descriptor / Wrong medium type

A nice and mysterious title. As usual. ๐Ÿ˜. But you know what I’m speaking about. I had one GPFS share that crashed due to a failing NVMe disk. The GPFS share was composed of several disks, and configured without safety, as a scratch disk. Being naive, I thought that simply replacing the failed disk by a disk that will call here Disk it and rebooting everything should bring my GPFS share back. It didn’t work, and I have learned some new things that I’d like to show you.

To recover my shared gpfsshare after the replacing of the new disk, first I’ve tried changing the StanzaFile disk names and recreate the NSD disks by running mmcrnsd -F StanzaFile. I could join the new disk Disk but the share was not yet usable.

Then I tried to change the mounting point of gpfsshare from /mountpoint to /newmountpoint

root@gpfs ~ ## > mmchfs gpfsshare -T /newmountpoint
Verifying file system configuration information ...
Disk Disk: Incompatible file system descriptor version or not formatted.
Failed to read a file system descriptor.
Wrong medium type
mmchfs: Failed to collect required file system attributes.
mmchfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

Next thing I thought on doing is deleting Disk from gpfsshare. That way, I would end up with a GPFS share, smaller than the original, but functional.

root@gpfs ~ ## > mmdeldisk gpfsshare Disk
Verifying file system configuration information โ€ฆ
Too many disks are unavailable.
Some file system data are inaccessible at this time.
Check error log for additional information.
Too many disks are unavailable.
Some file system data are inaccessible at this time.
mmdeldisk: Failed to collect required file system attributes.
mmdeldisk: Unexpected error from reconcileSdrfsWithDaemon.
Return code: 1
mmdeldisk: Attention:
File system gpfsshare may have some disks
that are in a non-ready state.
Issue the command:
mmcommon recoverfs scratch
mmdeldisk: Command failed.
Examine previous error messages to determine cause.

Let’s list our NSD disks to see what we have. We can list with mmlsnsd -X or without the argument (-X). This is my output (edited):

root@gpfs ~ ## > mmlsnsd

File system | Disk name | NSD servers
-----------------------------------------------
gpfsshare Disk node1.domain.org
gpfsshare Disk_old1 node2.domain.org
gpfsshare Disk_old2 node3.domain.org
(free disk) Disk_A node4.domain.org
(free disk) Disk_B node5.domain.org
(free disk) Disk_C node6.domain.org

Since everything is looking awful here, I will delete my gpfsshare filesystem and make it new. Actually I need to force the deletion. Let me show you.

root@gpfs ## > mmdelfs gpfsshare
Disk Disk: Incompatible file system descriptor version or not formatted.
Failed to read a file system descriptor.
Wrong medium type
mmdelfs: tsdelfs failed.
mmdelfs: Command failed. Examine previous error messages to determine cause.
root@gpfs ## > mmdelfs gpfsshare -p
Disk Disk: Incompatible file system descriptor version or not formatted.
Failed to read a file system descriptor.
Wrong medium type
mmdelfs: Attention: Not all disks were marked as available.
mmdelfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

We managed to delete the filesystem! But what happened with our NFS disks? Let’s use mmlsnsd to see where they stand:

root@gpfs ~ ## > mmlsnsd

File system | Disk name | NSD servers
-----------------------------------------------
(free disk) Disk node1.domain.org
(free disk) Disk_old1 node2.domain.org
(free disk) Disk_old2 node3.domain.org
(free disk) Disk_A node4.domain.org
(free disk) Disk_B node5.domain.org
(free disk) Disk_C node6.domain.org

So the disks are there. The filesystem gpfsshare is gone so they are marked as belonging to a free disk filesystem. Let’s then delete the NSD disks. We need to do it one by one. I show the output for the disk Disk.

root@gpfs ## > mmdelnsd Disk
mmdelnsd: Processing disk Disk
mmdelnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

Once we removed all our disks (check with mmlsnsd) we can add them again using the original StanzaFile (mmcrnsd -F StanzaFile). If you have already a disk processed, don’t worry: the mmcrnsd disk will anyway work. A standard output for our disks will be like this:

root@gpfs ## > mmcrnsd -F StanzaFile
mmcrnsd: Processing disk Disk_A
mmcrnsd: Disk name Disk_A is already registered for use by GPFS.
mmcrnsd: Processing disk Disk_B
mmcrnsd: Disk name Disk_B is already registered for use by GPFS.
mmcrnsd: Processing disk Disk_C
mmcrnsd: Disk name Disk_C is already registered for use by GPFS.
mmcrnsd: Processing disk Disk
mmcrnsd: Processing disk Disk_old1
mmcrnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

Time to re-create the gpfs filesystem. We can call it the same way as before ๐Ÿ˜‰, if we like. It works now! I have changed nothing with respect to the command I used originally to create my gpfs share gpfsshare. This one:

mmcrfs gpfsshare -F StanzaFile -T /mountpoint -m 2 -M 3 -i 4096 -A yes -Q no -S relatime -E no –version=5.X.Y.Z

If you get an error like this:

Unable to open disk 'Disk_A' on node node4.domain.org
No such device

Check that the gpfs is running on the corresponding node (in this example it’s node4.domain.org) and run again the mmcrfs.

I hope you have learned something already. But for my notes, first delete the filesystem (mmdelfs gpfsshare -p), second delete the nsf disks (mmdelnsd Disk), then add the disks again (mmcrnsd -F StanzaFile), then create a new filesystem (mmcrfs). Take care!

Kubernetes metric-server not going to READY 1/1 on CentOS 7.X

My kubernetes dashboard seemed to have lost the metrics at one point. And this is bad, because I do need to monitor the resources, since my cluster is baremetal and I don’t plug any web service. The symptoms are below. I deploy the metric server and check the status:

root@kube ## > kubectl apply -f \
https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
root@kube ## > kubectl get deployments --all-namespaces
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default ubuntu-desktop 1/1 1 1 3d23h
development snowflake 2/2 2 2 4d22h
kube-system coredns 2/2 2 2 39d
kube-system metrics-server 0/1 1 0 2s
kube-system skooner 1/1 1 1 39m
portainer portainer 1/1 1 1 4d20h
production cattle 5/5 5 5 4d22h

Also, on the dashboard Workload Status the service fails. I’ve waited long enough but my graphs don’t appear. First I uninstall the dashboard.

root@kube  ## > kubectl delete -f \
https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

serviceaccount "metrics-server" deleted
clusterrole.rbac.authorization.k8s.io "system:aggregated-metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "system:metrics-server" deleted
rolebinding.rbac.authorization.k8s.io "metrics-server-auth-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "metrics-server:system:auth-delegator" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-server" deleted
service "metrics-server" deleted
deployment.apps "metrics-server" deleted
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted

Then I make a copy of the metric server manifest locally (open the link on the browser, copy it as a yaml manifest) and edit it. The args (lines 129 to 140) look now like this:

spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls=true
image: k8s.gcr.io/metrics-server/metrics-server:v0.5.0
imagePullPolicy: IfNotPresent
livenessProbe:

The main change is in red. Of course, careful with the indentation! Because you know, yaml, ๐Ÿ˜๐Ÿ˜. Now I deploy the modified manifest

root@kube ## > kubectl apply -f kube-metric-server-new.yaml

and wait for my graphs to start appearing on the dashboard (~ 15 minutes). Here the issue thread from github. If I have learned something is that one needs to have local copies of the manifests, just in case. ๐Ÿ˜‰๐Ÿ˜‰. BONUS: documentation about deployment on Kubernetes and the Kubernetes cheatsheet. Because I may have saved you one or two google searchs.

Portainer on Kubernetes for CentOS 7.X

I already installed Portainer as a docker and I’m very happy about it. Is it the same sensation on a Kubernetes cluster? I need to deploy it to see it ๐Ÿ˜‰. The official How-To is here. This my experience with it.

First I stomp over the pre-req Note. Since I have a baremetal Kubernetes cluster (that is, something I installed) I forgot to define the kubernetes storage classes. Here you have the kubernetes storage classes documentation. As you can expect, since it is a kubernetes feature, you can connect your cluster to a wide variety of storages, cloudly or not. I don’t want to speak about what I don’t know so I’ll simply go ahead and add from my Kubernetes Dashboard the yaml files for a storage class sc and for a persistent volume pv. Just press the “+” on the web and add to the yaml the next:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

This should create the sc called “local-storage“. I didn’t change a comma from the documentation. We can see it by typing kubeclt get sc. This is my output

root@kube ## > kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage (default) kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 13m

Sorry but there’s no way to format this right ๐Ÿ˜๐Ÿ˜. We will need also a local persistent volume. This is my yaml.

apiVersion: v1
kind: PersistentVolume
metadata:
name: kube-local-pv
spec:
capacity:
storage: 500Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /my/storage/
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- kube

I didn’t change so much with respect to the kubernetes local persistent volume documentation. Also I add it through the web, although you should have already realized how to do it from the command line.ย Now that we have our storages, we can go ahead and deploy. I try first through helm, but somehow it doesn’t seem to work. Anyway, here’s my output:

root@kube ~ ## > helm install -n portainer portainer portainer/portainer --set persistence.storageClass=kube-local-pv
NAME: portainer
LAST DEPLOYED: Thu Aug 26 14:16:11 2021
NAMESPACE: portainer
STATUS: deployed
REVISION: 1
NOTES:
Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace portainer -o jsonpath="{.spec.ports[0].nodePort}" services portainer)
export NODE_IP=$(kubectl get nodes --namespace portainer -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT

Again, sorry for the editing. The above command is not like in the tutorial due to the changes in helm v3. I guess the kubernetes world is evolving pretty quickly, so to say. I don’t pass the namespace but I create it through command line.

kubectl create -f portainer.yaml

My portainer.yaml looks like this:

apiVersion: v1
kind: Namespace
metadata:
name: portainer

Of course it will work also if you add that through the dashboard. Anyhow. It didn’t work with helm (the pod was always pending resources) so I went for a YAML manifest install. First we undo the deployment by deleten the namespace and the cluster role. We can do that from the dasboard or via command line. The command line cleanup looks like this:

root@kube ~ ## > kubectl delete namespace portainer
namespace "portainer" deleted
root@kube ~ ## > kubectl delete clusterrolebinding portainer
clusterrolebinding.rbac.authorization.k8s.io "portainer" deleted

Then we deploy as NodePort. This is my output:

root@kube ~ ## > kubectl apply -n portainer -f https://raw.githubusercontent.com/portainer/k8s/master/deploy/manifests/portainer/portainer.yaml
namespace/portainer created
serviceaccount/portainer-sa-clusteradmin created
persistentvolumeclaim/portainer created
clusterrolebinding.rbac.authorization.k8s.io/portainer created
service/portainer created
deployment.apps/portainer created

After a few second, I can access to my portainer gui through the address kube:30777. Everything is pretty similar to the docker version, so โค๏ธโค๏ธ success!

Rootless docker error on CentOS 7: failed to mount overlay: operation not permitted storage-driver=overlay2

While trying a rootless docker on my servers, I found a lot of issues. They reccomend to use an Ubuntu kernel, but I use CentOS 7.X. So I need to stick with it. The prerequisites are fine – I have newuidmap and newgidmap and enough subordinates. This is how it looks like when I run the rootless setup script as an user user.

user@server ~ $ > dockerd-rootless-setuptool.sh install
[ERROR] Missing system requirements.
[ERROR] Run the following commands to
[ERROR] install the requirements and run this tool again.

########## BEGIN ##########
sudo sh -eux <<EOF
# Set user.max_user_namespaces
cat <<EOT > /etc/sysctl.d/51-rootless.conf
user.max_user_namespaces = 28633
EOT
sysctl --system
# Add subuid entry for user
echo "user:100000:65536" >> /etc/subuid
# Add subgid entry for user
echo "user:100000:65536" >> /etc/subgid
EOF
########## END ##########

We go as root and cut and copy the thing above between the #. This is the output, edited.

root@server ~ ## > cut-and-copy-of-the-thing-above
+ cat
+ sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
kernel.kptr_restrict = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/51-rootless.conf ...
user.max_user_namespaces = 28633
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.conf ...
+ echo user:100000:65536
+ echo user:100000:65536
root@server ~ ## > ########## END ##########

Time to try again. The result gives no error but is not like in the tutorial. Here you have it.

user@server ~ $ > dockerd-rootless-setuptool.sh install
[INFO] systemd not detected, dockerd-rootless.sh
needs to be started manually:
PATH=/usr/bin:/sbin:/usr/sbin:$PATH dockerd-rootless.sh
[INFO] Creating CLI context "rootless"
Successfully created context "rootless"
[INFO] Make sure the following environment variables
are set (or add them to ~/.bashrc):
export PATH=/usr/bin:$PATH
export DOCKER_HOST=unix:///run/user/3201/docker.sock
user@server ~ $ >

So what does it mean to start it manually? After reading this bug report, I decide to try to run it with the experimental tag and specifying the storage driver. This is my output, as usual, edited for a proper reading. Important messages in blue, comments in cursive, errors in red.

user@server ~ $ > dockerd-rootless.sh \
--experimental --storage-driver overlay2
+ case "$1" in
+ '[' -w /run/user/USERID ']'
+ '[' -w /home/user ']'
--> some user-dependent messages...
+ exec dockerd --experimental --storage-driver overlay2
INFO[] Starting up
WARN[] Running experimental build
WARN[] Running in rootless mode.
This mode has feature limitations.
INFO[] Running with RootlessKit integration
...more messages here, loading plugins...
INFO[] skip loading plugin "io.containerd.snapshotter.v1.aufs"...
error="aufs is not supported: skip plugin"
type=io.containerd.snapshotter.v1
INFO[] loading plugin "io.containerd.snapshotter.v1.devmapper"...
type=io.containerd.snapshotter.v1
WARN[] failed to load plugin e
rror="devmapper not configured"

INFO[] loading plugins..
INFO[] skip loading plugin "io.containerd.snapshotter.v1.zfs"...
error="path must be a zfs : skip plugin"
type=io.containerd.snapshotter.v1
WARN[] could not use snapshotter devmapper
in metadata plugin
error="devmapper not configured"

INFO[] metadata content store policy set policy=shared
INFO[] loading a lot of plugins sucessfully...
...more messages here, loading plugins...
INFO[] serving... address=/run/user/USERID/docker/containerd/sockets
INFO[] serving...
INFO[] containerd successfully booted in 0.033234s
WARN[] Could not set may_detach_mounts kernel parameter
error="error opening may_detach_mounts kernel config file:
open /proc/sys/fs/may_detach_mounts: permission denied"

INFO[] parsed scheme: "unix" module=grpc
...more messages here...
INFO[] ClientConn switching balancer to "pick_first" module=grpc
ERRO[] failed to mount overlay:
operation not permitted storage-driver=overlay2
INFO[] stopping event stream following graceful shutdown
error="context canceled"
module=libcontainerd namespace=plugins.moby
failed to start daemon:
error initializing graphdriver: driver not supported

[rootlesskit:child ] error:
command [/usr/bin/dockerd-rootless.sh
--experimental --storage-driver overlay2]
exited: exit status 1

[rootlesskit:parent] error: child exited: exit status 1

What do I get from the above run? There are warnings on zfs, aufs, and finally overlay2 so it looks like there’s some kind of problem with the storage driver. You can get also a dark failed to register layer message or an error creating overlay mount. It makes sense, since I’m coming from a fully working root install. I try once more without the storage driver option, and this is the (interesting part of) the output.

ERRO[] failed to mount overlay: 
operation not permitted storage-driver=overlay2
ERRO[] AUFS cannot be used in non-init user namespace
storage-driver=aufs
ERRO[] failed to mount overlay: operation not permitted
storage-driver=overlay
INFO[] Attempting next endpoint for pull after error:
failed to register layer:
ApplyLayer exit status 1 stdout:
stderr: open /root/.bash_logout: permission denied

So if you don’t give a storage option, it tries them all. Mystery solved, I guess. You can have a look to the available overlayfs documentation (covering overlay and overlay2). But in short, the docker daemon running under user doesn’t manage to access to the storage drivers. Let’s have a look onto the docker storage options. Some documentation first. We know how to change the directory to store containers and images in Docker. In my CentOS 7.X, I see my daemon runs overlay2 and indeed the images after dowloaded are stored on /var/lib/docker/overlay2. I can change the docker image installation directory by editing /etc/docker/daemon.json. I add something like this.

{
"data-root": "/extrahd/docker",
"storage-driver": "overlay2"
}

I then clean up by docker system prune -a and restart my docker daemon still as root to be sure the newly downloaded images end up on /extrahd/docker. As expected. ๐Ÿ˜‰. In principle the given location cannot be a GPFS or a CIFS mounted folder, or I end up getting all the driver errors.

Will this work for on rootless mode? In my case,it was not possible until I did the same trick as for root. So one needs to configure the user docker daemon. For my user user, it should be located on

 /home/user/.config/docker/daemon.json 

Remember, of course, that the storage must be writable by the user user ๐Ÿ˜‰๐Ÿ˜‰. I hope now you can run docker rootless on CentOS 7.X as I can! ๐Ÿค˜๐Ÿ˜Š.

Bonus: a docker storage using a bind mount (not tested) and how to control the docker with systemd, and the full docker daemon configuration file documentation.

The 3-2-1 backup rule

I’m reading now the book Kubernetes Backup & Recovery for dummies (Kasten by Veeam Special edition) that I got for free after my Cloud Native Days with Kubernetes conference. First I need to say I’m not a native cloud inhabitant, but I hope to be ready to live in the clouds when the time comes. So I can’t comment a lot about the book as a whole, but it’s definitely a very good overview of the problem of the backup and recovery. There’s one topic that I think it goes beyond kubernetes that I want to annotate here.

It’s named as a timeless rule against a failure scenario, so I guess it’s OK to copy it here. It is called, you guessed it right, the 3-2-1 backup rule. This rule answers the two questions.

  • How many backup files should I have?
  • where should I store them?

Answers:

ยป 3: Have at least three copies of your data.

ยป 2: Store the copies on two different media.

ยป 1: Keep one backup copy offsite.

It may sound easy to achieve, but it is not when we have petabytes of data. We do have an HPSS with snapshots (one) and a TSM server for the home folders, but we can’t ask everyone to keep an offline copy of their data. I do ask the people to follow this rule (at least 3 copies) of their important documents (papers, etc). So I’m going to add a zero rule here: select what you want to backup. And do not trust the black box on this one. You must back up what you want to keep. Just in case everything else fails ๐Ÿ˜‰.

Portainer, a docker GUI in a docker, on CentOS 7.X

The Portainer web. As taken from this tutorial.

It’s been a while since I started with the dockers and the kubernetes but so far I didn’t show you a docker management solution, only a docker usage cheat sheet. Well, forget about that, now I give you Portainer, a web that will allow you to check your images, volumes, and containers. Of course, running as a docker.

The image above I’ve taken from this portainer installation tutorial for Ubuntu. It includes a docker installation, but I expect you have already have dockers running. Anyway, once with dockers, the install is pretty simple. Just like this:

docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock portainer/portainer

After that, you should be able to access to the web (machinename:9000) and finish it up by creating the initial admin password. So far I’m very satisifed with the experience, but if you are not, you can check this comparison of docker GUIs. What’s next? Isolation, security and kubernetes integration. I’ll keep you posted…