A new era is about to begin. As you may have gathered from my previous texts, I’m a CentOS 7.X user. As a such, I’m being forced by obsolescence to move to a new OS. That will probably be some kind of Ubuntu. All our clients need to be connected to our GPFS cluster, so in the next lines you are going to read how it was done in my case.
We log in into the machine. We are going to use Spectrum_Scale_Standard-5.1.5.1-x86_64-Linux-install provided by IBM. We simply run it.
# ./Spectrum_Scale_Standard-5.1.5.1-x86_64-Linux-install
Extracting License Acceptance Process Tool to /usr/lpp/mmfs/5.1.5.1 ...
tail -n +660 ./Spectrum_Scale_Standard-5.1.5.1-x86_64-Linux-install | tar -C /usr/lpp/mmfs/5.1.5.1 -xvz --exclude=installer --exclude=*_rpms --exclude=*_debs --exclude=*rpm --exclude=*tgz --exclude=*deb --exclude=*tools* 1> /dev/null
Installing JRE ...
If directory /usr/lpp/mmfs/5.1.5.1 has been created or was previously created during another extraction,
.rpm, .deb, and repository related files in it (if there were) will be removed to avoid conflicts with the ones being extracted.
removed '/usr/lpp/mmfs/5.1.5.1/hdfs_rpms/rhel/hdfs_3.1.1.x/repodata/repomd.xml.asc'
... a lot of other stuff being removed...
tail -n +660 ./Spectrum_Scale_Standard-5.1.5.1-x86_64-Linux-install | tar -C /usr/lpp/mmfs/5.1.5.1 --wildcards -xvz ibm-java*tgz 1> /dev/null
tar -C /usr/lpp/mmfs/5.1.5.1/ -xzf /usr/lpp/mmfs/5.1.5.1/ibm-java*tgz
Invoking License Acceptance Process Tool ...
/usr/lpp/mmfs/5.1.5.1/ibm-java-x86_64-80/jre/bin/java -cp /usr/lpp/mmfs/5.1.5.1/LAP_HOME/LAPApp.jar com.ibm.lex.lapapp.LAP -l /usr/lpp/mmfs/5.1.5.1/LA_HOME -m /usr/lpp/mmfs/5.1.5.1 -s /usr/lpp/mmfs/5.1.5.1
The we get a po-up that is asking you to accept the terms. We say “yes” (do we have an option?) and the installation goes ahead.
License Agreement Terms accepted.
Extracting Product RPMs to /usr/lpp/mmfs/5.1.5.1 ...
tail -n +660 ./Spectrum_Scale_Standard-5.1.5.1-x86_64-Linux-install | tar -C /usr/lpp/mmfs/5.1.5.1 --wildcards -xvz Public_Keys ansible-toolkit ganesha_debs/ubuntu/ubuntu20 ganesha_debs/ubuntu/ubuntu22 gpfs_debs/ubuntu/ubuntu20 gpfs_debs/ubuntu/ubuntu22 hdfs_rpms/rhel/hdfs_3.1.1.x hdfs_rpms/rhel/hdfs_3.2.2.x hdfs_rpms/rhel/hdfs_3.3.x smb_debs/ubuntu/ubuntu20 smb_debs/ubuntu/ubuntu22 zimon_debs/ubuntu/ubuntu20 zimon_debs/ubuntu/ubuntu22 ganesha_rpms/rhel7 ganesha_rpms/rhel8 ganesha_rpms/sles15 gpfs_rpms/rhel7 gpfs_rpms/rhel8 gpfs_rpms/sles15 object_rpms/rhel8 smb_rpms/rhel7 smb_rpms/rhel8 smb_rpms/sles15 tools/repo zimon_debs/ubuntu zimon_rpms/rhel7 zimon_rpms/rhel8 zimon_rpms/sles15 gpfs_debs gpfs_rpms manifest 1> /dev/null
- Public_Keys
- ansible-toolkit
- ganesha_debs/ubuntu/ubuntu20
- ganesha_debs/ubuntu/ubuntu22
- gpfs_debs/ubuntu/ubuntu20
- gpfs_debs/ubuntu/ubuntu22
- hdfs_rpms/rhel/hdfs_3.1.1.x
- hdfs_rpms/rhel/hdfs_3.2.2.x
- hdfs_rpms/rhel/hdfs_3.3.x
- smb_debs/ubuntu/ubuntu20
- smb_debs/ubuntu/ubuntu22
- zimon_debs/ubuntu/ubuntu20
- zimon_debs/ubuntu/ubuntu22
- ganesha_rpms/rhel7
- ganesha_rpms/rhel8
- ganesha_rpms/sles15
- gpfs_rpms/rhel7
- gpfs_rpms/rhel8
- gpfs_rpms/sles15
- object_rpms/rhel8
- smb_rpms/rhel7
- smb_rpms/rhel8
- smb_rpms/sles15
- tools/repo
- zimon_debs/ubuntu
- zimon_rpms/rhel7
- zimon_rpms/rhel8
- zimon_rpms/sles15
- gpfs_debs
- gpfs_rpms
- manifest
Removing License Acceptance Process Tool from /usr/lpp/mmfs/5.1.5.1 ...
rm -rf /usr/lpp/mmfs/5.1.5.1/LAP_HOME
/usr/lpp/mmfs/5.1.5.1/LA_HOME
Removing JRE from /usr/lpp/mmfs/5.1.5.1 ...
rm -rf /usr/lpp/mmfs/5.1.5.1/ibm-java*tgz
==================================================================
Product packages successfully extracted to /usr/lpp/mmfs/5.1.5.1
Cluster installation and protocol deployment
To install a cluster or deploy protocols with the
IBM Spectrum Scale Installation Toolkit:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale -h
To install a cluster manually: Use the GPFS packages located
within /usr/lpp/mmfs/5.1.5.1/gpfs_<rpms/debs>
To upgrade an existing cluster using the
IBM Spectrum Scale Installation Toolkit:
1) Review and update the config:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale
config update
2) Update the cluster configuration to reflect
the current cluster config:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale
config populate -N <node>
3) Use online or offline upgrade depending on your requirements:
- Run the online rolling upgrade:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale
upgrade -h
- Run the offline upgrade:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale
upgrade config offline -N;
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale
upgrade run
You can also run the parallel offline upgrade
to upgrade all nodes parallely after shutting down GPFS
and stopping protocol services on all nodes.
You can run the parallel offline upgrade
on all nodes in the cluster, not on a subset of nodes.
To add nodes to an existing cluster using the
IBM Spectrum Scale Installation Toolkit:
1) Add nodes to the cluster definition file:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale node add -h
2) Install IBM Spectrum Scale on the new nodes:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale install -h
3) Deploy protocols on the new nodes:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale deploy -h
To add NSDs or file systems to an existing cluster
using the IBM Spectrum Scale Installation Toolkit:
1) Add NSDs or file systems to the cluster definition:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale nsd add -h
2) Install the NSDs or file systems:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale install -h
To update the cluster definition to reflect
the current cluster config examples:
/usr/lpp/mmfs/5.1.5.1/ansible-toolkit/spectrumscale config populate -N <node>
1) Manual updates outside of the installation toolkit
2) Sync the current cluster state
to the installation toolkit prior to upgrade
3) Switching from a manually managed cluster
to the installation toolkit
==========================================================
To get up and running quickly, consult the
IBM Spectrum Scale Protocols Quick Overview:
https://www.ibm.com/docs/en/STXKQY_5.1.5/pdf/scale_povr.pdf
==========================================================
For convenience and later use we copy the debian packages to our current folder before installing them.
# cp -Rav /usr/lpp/mmfs/5.1.5.1/gpfs_debs/ .
Time to install. We use dpkg. Other options are available, but this one is native.
# dpkg -i gpfs.base_5.1.5-1_amd64.deb \
gpfs.compression_5.1.5-1_amd64.deb \
gpfs.docs_5.1.5-1_all.deb \
gpfs.gpl_5.1.5-1_all.deb \
gpfs.gskit_8.0.55-19.1_amd64.deb \
gpfs.license.std_5.1.5-1_amd64.deb \
gpfs.msg.en-us_5.1.5-1_all.deb
(Reading database ... 206152 files and directories currently installed.)
Preparing to unpack gpfs.base_5.1.5-1_amd64.deb ...
Unpacking gpfs.base (5.1.5-1) over (5.1.5-1) ...
Preparing to unpack gpfs.compression_5.1.5-1_amd64.deb ...
Unpacking gpfs.compression (5.1.5-1) over (5.1.5-1) ...
Preparing to unpack gpfs.docs_5.1.5-1_all.deb ...
Unpacking gpfs.docs (5.1.5-1) over (5.1.5-1) ...
Preparing to unpack gpfs.gpl_5.1.5-1_all.deb ...
make[1]: Entering directory '/usr/lpp/mmfs/src'
rm -rf /usr/lpp/mmfs/src/include /usr/lpp/mmfs/src/bin /usr/lpp/mmfs/src/lib
mkdir /usr/lpp/mmfs/src/include /usr/lpp/mmfs/src/bin /usr/lpp/mmfs/src/lib
rm -f //usr/lpp/mmfs/src/gpl-linux/gpl_kernel.tmp.ver
cleaning (/usr/lpp/mmfs/src/ibm-kxi)
make[2]: Entering directory '/usr/lpp/mmfs/src/ibm-kxi'
rm -f install.he; \
for i in cxiTypes.h cxiSystem.h cxi2gpfs.h cxiVFSStats.h cxiCred.h cxiIOBuffer.h cxiSharedSeg.h cxiMode.h Trace.h cxiMmap.h cxiAtomic.h cxiTSFattr.h cxiAclUser.h cxiLinkList.h cxiDmapi.h LockNames.h lxtrace.h cxiGcryptoDefs.h cxiSynchNames.h cxiMiscNames.h cxiPMem.h DirIds.h; do \
(set -x; rm -f -r /usr/lpp/mmfs/src/include/cxi/$i) done
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiTypes.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiSystem.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxi2gpfs.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiVFSStats.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiCred.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiIOBuffer.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiSharedSeg.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiMode.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/Trace.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiMmap.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiAtomic.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiTSFattr.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiAclUser.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiLinkList.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiDmapi.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/LockNames.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/lxtrace.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiGcryptoDefs.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiSynchNames.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiMiscNames.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/cxiPMem.h
+ rm -f -r /usr/lpp/mmfs/src/include/cxi/DirIds.h
make[2]: Leaving directory '/usr/lpp/mmfs/src/ibm-kxi'
cleaning (/usr/lpp/mmfs/src/ibm-linux)
make[2]: Entering directory '/usr/lpp/mmfs/src/ibm-linux'
rm -f install.he; \
for i in cxiTypes-plat.h cxiSystem-plat.h cxiIOBuffer-plat.h cxiSharedSeg-plat.h cxiMode-plat.h Trace-plat.h cxiAtomic-plat.h cxiMmap-plat.h cxiVFSStats-plat.h cxiCred-plat.h cxiDmapi-plat.h; do \
(set -x; rm -rf /usr/lpp/mmfs/src/include/cxi/$i) done
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiTypes-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiSystem-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiIOBuffer-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiSharedSeg-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiMode-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/Trace-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiAtomic-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiMmap-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiVFSStats-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiCred-plat.h
+ rm -rf /usr/lpp/mmfs/src/include/cxi/cxiDmapi-plat.h
make[2]: Leaving directory '/usr/lpp/mmfs/src/ibm-linux'
cleaning (/usr/lpp/mmfs/src/gpl-linux)
make[2]: Entering directory '/usr/lpp/mmfs/src/gpl-linux'
Pre-kbuild step 1...
/usr/bin/make -C /lib/modules/5.15.0-60-generic/build M=/usr/lpp/mmfs/src/gpl-linux clean
make[3]: Entering directory '/usr/src/linux-headers-5.15.0-60-generic'
CLEAN /usr/lpp/mmfs/src/gpl-linux
CLEAN /usr/lpp/mmfs/src/gpl-linux/Module.symvers
make[3]: Leaving directory '/usr/src/linux-headers-5.15.0-60-generic'
rm -f -f /lib/modules/`cat //usr/lpp/mmfs/src/gpl-linux/gpl_kernel.tmp.ver`/extra/tracedev.ko
rm -f -f /lib/modules/`cat //usr/lpp/mmfs/src/gpl-linux/gpl_kernel.tmp.ver`/extra/mmfslinux.ko
rm -f -f /lib/modules/`cat //usr/lpp/mmfs/src/gpl-linux/gpl_kernel.tmp.ver`/extra/mmfs26.ko
rm -f -f /usr/lpp/mmfs/src/../bin/lxtrace-`cat //usr/lpp/mmfs/src/gpl-linux/gpl_kernel.tmp.ver`
rm -f -f /usr/lpp/mmfs/src/../bin/kdump-`cat //usr/lpp/mmfs/src/gpl-linux/gpl_kernel.tmp.ver`
rm -f -f *.o .depends .*.cmd *.ko *.a *.mod.c core *_shipped *map *mod.c.saved *.symvers *.ko.ver ./*.ver install.he
rm -f -rf .tmp_versions kdump-kern-dwarfs.c
rm -f -f gpl-linux.trclst kdump lxtrace
rm -f -rf usr
make[2]: Leaving directory '/usr/lpp/mmfs/src/gpl-linux'
make[1]: Leaving directory '/usr/lpp/mmfs/src'
Unpacking gpfs.gpl (5.1.5-1) over (5.1.5-1) ...
Preparing to unpack gpfs.gskit_8.0.55-19.1_amd64.deb ...
Unpacking gpfs.gskit (8.0.55-19.1) over (8.0.55-19.1) ...
Preparing to unpack gpfs.license.std_5.1.5-1_amd64.deb ...
Unpacking gpfs.license.std (5.1.5-1) over (5.1.5-1) ...
Preparing to unpack gpfs.msg.en-us_5.1.5-1_all.deb ...
Unpacking gpfs.msg.en-us (5.1.5-1) over (5.1.5-1) ...
Setting up gpfs.base (5.1.5-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/mmautoload.service ? /lib/systemd/system/mmautoload.service.
Created symlink /etc/systemd/system/multi-user.target.wants/mmccrmonitor.service ? /lib/systemd/system/mmccrmonitor.service.
Setting up gpfs.compression (5.1.5-1) ...
Setting up gpfs.docs (5.1.5-1) ...
Setting up gpfs.gpl (5.1.5-1) ...
Setting up gpfs.gskit (8.0.55-19.1) ...
Setting up gpfs.license.std (5.1.5-1) ...
Setting up gpfs.msg.en-us (5.1.5-1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Now we go to the GPFS quorum-manager and try to join the node. First we try that we can do passwordless ssh to it. That we can. Let’s call our new client ubuntu00.domain.org. I add it to the current cluster like this:
@quorum # > mmaddnode -N ubuntu00.domain.org
@quorum # > mmchlicense client --accept -N ubuntu00.domain.org
We come back now to our client ubuntu00 and test to start the gpfs. It doesn’t start, but it’s fine. We build the kernel extension then:
# /usr/lpp/mmfs/bin/mmbuildgpl
--------------------------------------------------------
mmbuildgpl: Building GPL (5.1.5.1) module begins at DATE.
--------------------------------------------------------
Verifying Kernel Header...
kernel version = 51500060 (515000060000000, 5.15.0-60-generic, 5.15.0-60)
module include dir = /lib/modules/5.15.0-60-generic/build/include
module build dir = /lib/modules/5.15.0-60-generic/build
kernel source dir = /usr/src/linux-5.15.0-60-generic/include
Found valid kernel header file under /lib/modules/5.15.0-60-generic/build/include
Getting Kernel Cipher mode...
Will use skcipher routines
Verifying Compiler...
make is present at /bin/make
cpp is present at /bin/cpp
gcc is present at /bin/gcc
g++ is present at /bin/g++
ld is present at /bin/ld
make World ...
make InstallImages ...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at DATE.
--------------------------------------------------------
Now gpfs start without issues. I’m being lucky lately, everything I try works out of the box. Maybe because I’m using a modern OS. We’ll see how long until I find a big stone… π¦