Install GPFS on CentOS 7

This is just a recipe. It may not work for another system. First I need to install the infiniband drivers. I downloaded the latest drivers from the mellanox page. Previously I installed all the packages needed to compile them. Anyway, the installation will complain if something is missing, and it will ask you to install it. I am installing them over a minimum CentOS 7 install, after the system update and installing my missing packages.

Here is an extract of the installation dump:

root@server ~/bin ## > 
tar -xvzf MLNX_OFED_LINUX-3.4- 
root@server ~/bin/MLNX_OFED_LINUX-3.4- ## > 
./mlnxofedinstall --add-kernel-support -vv
...bla bla bla...
Below is the list of MLNX_OFED_LINUX packages that you have chosen
(some may have been added by the installer 
due to package dependencies): the list...
This program will install the MLNX_OFED_LINUX package 
on your machine. Note that all other Mellanox, OEM, OFED, 
or Distribution IB packages will be removed.
... here the installation is done, packages are verified also ...
- The FCA Manager and FCA MPI Runtime library are 
installed in /opt/mellanox/fca directory.
- The FCA Manager will not be started automatically. 
- To start FCA Manager now, type:
 /etc/init.d/fca_managerd start
- There should be single process of FCA Manager running per fabric.
- To start FCA Manager automatically after boot, type:
 /etc/init.d/fca_managerd install_service
- Check /opt/mellanox/fca/share/doc/fca/README.txt 
for quick start instructions.
... and now more packages are updated or installed ...
Verifying mpitests_openmpi-3.2.18-0cdeb98.34100.x86_64
Device (03:00.0):
 03:00.0 Network controller: 
Mellanox Technologies MT27500 Family [ConnectX-3]
 Link Width: x8
 PCI Link Speed: 8GT/s
Installation finished successfully.
Running: FW_UPDATE_FLAGS='--log 
...bla bla bla...
Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf
Attempting to perform Firmware update...
...bla bla bla...
Querying Mellanox devices firmware ...
Device #1:
Device Type: ConnectX3
 Part Number: XXXX
 Description: ConnectX-3 VPI adapter card; 
single-port QSFP; FDR IB (56Gb/s) 
and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
 PSID: MT_1100120019
 PCI Device Name: 03:00.0
 Port2 MAC: XXXX
 Versions: Current Available 
 FW 2.36.5000 2.36.5150 
 PXE 3.4.0718 3.4.0740
Status: Update required
Found 1 device(s) requiring firmware update...
Device #1: Updating FW ... Done
Restart needed for updates to take effect.
Log File: /tmp/MLNX_OFED_LINUX-3.4-
Please reboot your system for the changes to take effect.
To load the new driver, run:
/etc/init.d/openibd restart

Note that it also updates the firmware 🙂

I then add the connection already using the nm-connection-editor, and reboot. After the reboot, I NFS-mount the folder (that is in alpha) where I store the gpfs packages

root@server ~ ## > mkdir /install
root@server ~ ## > mount alpha:/install /install
root@server ~ ## > cd /install/system/gpfs/gpfs_4.1.1.0/
root@server /install/system/gpfs/gpfs_4.1.1.0 ## > 
rpm -Uhv gpfs*-0*rpm
--> packages installed
root@server /install/system/gpfs/gpfs_4.1.1.0 ## > 
cd ../gpfs_4.1.1.8/
root@server /install/system/gpfs/gpfs_4.1.1.8 ## > 
rpm -Uhv gpfs*-8*rpm
--> more packages installed
root@server /install/system/gpfs/gpfs_4.1.1.8 ## > 
cd /usr/lpp/mmfs/src
root@server /usr/lpp/mmfs/src ## > 
LINUX_DISTRIBUTION=REDHAT_AS_LINUX make clean Autoconfig World rpm
rm -rf /usr/lpp/mmfs/src/include 
/usr/lpp/mmfs/src/bin /usr/lpp/mmfs/src/lib
...some stuff comes here...
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.5xF9Wr
+ umask 022
+ cd /root/rpmbuild/BUILD
+ /usr/bin/rm -rf /tmp/rpm
+ exit 0
--> now we can install the compiled version
root@server /usr/lpp/mmfs/src ## > rpm -Uvh 
--> done

Explanation: we install the rpms twice, then we configure them (in bold). Then I rsync the “old” server ssh keys, and mmfs (before the system was SL7) and reboot.

root@bkup /data_temp ## > 
rsync -av var/mmfs/ root@server:/var/mmfs/
root@bkup /data_temp ## >
rsync -av ssh/ root@server:/etc/ssh/ --delete-after --progress

I test that all is fine by login from the GPFS server itself.


About bitsanddragons

A traveller, an IT professional and a casual writer
This entry was posted in bits, centos, GPFS, linux, slurm. Bookmark the permalink.

One Response to Install GPFS on CentOS 7

  1. Pingback: Add a CentOS 7 node to a GPFS cluster | Bits and Dragons

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s