Install GPFS on CentOS 7

This is just a recipe. It may not work for another system. First I need to install the infiniband drivers. I downloaded the latest drivers from the mellanox page. Previously I installed all the packages needed to compile them. Anyway, the installation will complain if something is missing, and it will ask you to install it. I am installing them over a minimum CentOS 7 install, after the system update and installing my missing packages.

Here is an extract of the installation dump:

root@server ~/bin ## > 
tar -xvzf MLNX_OFED_LINUX-3.4-1.0.0.0-rhel7.2-x86_64.tgz 
root@server ~/bin/MLNX_OFED_LINUX-3.4-1.0.0.0-rhel7.2-x86_64 ## > 
./mlnxofedinstall --add-kernel-support -vv
...bla bla bla...
Below is the list of MLNX_OFED_LINUX packages that you have chosen
(some may have been added by the installer 
due to package dependencies):
...here the list...
This program will install the MLNX_OFED_LINUX package 
on your machine. Note that all other Mellanox, OEM, OFED, 
or Distribution IB packages will be removed.
... here the installation is done, packages are verified also ...
IMPORTANT NOTE:
===============
- The FCA Manager and FCA MPI Runtime library are 
installed in /opt/mellanox/fca directory.
- The FCA Manager will not be started automatically. 
- To start FCA Manager now, type:
 /etc/init.d/fca_managerd start
- There should be single process of FCA Manager running per fabric.
- To start FCA Manager automatically after boot, type:
 /etc/init.d/fca_managerd install_service
- Check /opt/mellanox/fca/share/doc/fca/README.txt 
for quick start instructions.
... and now more packages are updated or installed ...
Verifying mpitests_openmpi-3.2.18-0cdeb98.34100.x86_64
Device (03:00.0):
 03:00.0 Network controller: 
Mellanox Technologies MT27500 Family [ConnectX-3]
 Link Width: x8
 PCI Link Speed: 8GT/s
Installation finished successfully.
Running: FW_UPDATE_FLAGS='--log 
...bla bla bla...
Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf
Attempting to perform Firmware update...
...bla bla bla...
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
 Part Number: XXXX
 Description: ConnectX-3 VPI adapter card; 
single-port QSFP; FDR IB (56Gb/s) 
and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
 PSID: MT_1100120019
 PCI Device Name: 03:00.0
 Port1 GUID: XXXX
 Port2 MAC: XXXX
 Versions: Current Available 
 FW 2.36.5000 2.36.5150 
 PXE 3.4.0718 3.4.0740
Status: Update required
---------
Found 1 device(s) requiring firmware update...
Device #1: Updating FW ... Done
Restart needed for updates to take effect.
Log File: /tmp/MLNX_OFED_LINUX-3.4-1.0.0.0.400.logs/fw_update.log
Please reboot your system for the changes to take effect.
To load the new driver, run:
/etc/init.d/openibd restart

Note that it also updates the firmware 🙂

I then add the connection already using the nm-connection-editor, and reboot. After the reboot, I NFS-mount the folder (that is in alpha) where I store the gpfs packages

root@server ~ ## > mkdir /install
root@server ~ ## > mount alpha:/install /install
root@server ~ ## > cd /install/system/gpfs/gpfs_4.1.1.0/
root@server /install/system/gpfs/gpfs_4.1.1.0 ## > 
rpm -Uhv gpfs*-0*rpm
--> packages installed
root@server /install/system/gpfs/gpfs_4.1.1.0 ## > 
cd ../gpfs_4.1.1.8/
root@server /install/system/gpfs/gpfs_4.1.1.8 ## > 
rpm -Uhv gpfs*-8*rpm
--> more packages installed
root@server /install/system/gpfs/gpfs_4.1.1.8 ## > 
cd /usr/lpp/mmfs/src
root@server /usr/lpp/mmfs/src ## > 
LINUX_DISTRIBUTION=REDHAT_AS_LINUX make clean Autoconfig World rpm
rm -rf /usr/lpp/mmfs/src/include 
/usr/lpp/mmfs/src/bin /usr/lpp/mmfs/src/lib
...some stuff comes here...
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.5xF9Wr
+ umask 022
+ cd /root/rpmbuild/BUILD
+ /usr/bin/rm -rf /tmp/rpm
+ exit 0
--> now we can install the compiled version
root@server /usr/lpp/mmfs/src ## > rpm -Uvh 
/root/rpmbuild/RPMS/x86_64/gpfs.gplbin-3.10.0
-327.36.1.el7.x86_64-4.1.1-8.x86_64.rpm
--> done

Explanation: we install the rpms twice, then we configure them (in bold). Then I rsync the “old” server ssh keys, and mmfs (before the system was SL7) and reboot.

root@bkup /data_temp ## > 
rsync -av var/mmfs/ root@server:/var/mmfs/
root@bkup /data_temp ## >
rsync -av ssh/ root@server:/etc/ssh/ --delete-after --progress

I test that all is fine by login from the GPFS server itself.

Advertisements

About bitsanddragons

A traveller, an IT professional and a casual writer
This entry was posted in bits, centos, linux. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s