author：headsen chen
date： 2019-01-18 10:22:20
notice：created by headsen chen himself and not allowed to copy, or you will count law question!
版本环境：centos6.8 ，64位，内核：2.6.32
1，配置网卡：
在新卡装上机器，接收光纤，两根线都有接，而且是反接的方式接，接通后，光纤灯会亮
2，安装软件RDMA的方式安装，编译内核和用户态，重启进入新内核4.7
3，安装驱动：
正常的kernel安装方法（2.6的内核）
# /mnt/mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64/install
这里必须采用这种，因为是新内核4.7
# tar fx mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64.tgz
# cd mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64
# ./install --add-kernel-support --skip-repo Logs dir: /tmp/mlnx-en.28728.logs
General log file: /tmp/mlnx-en.28728.logs/general.log
Verifying KMP rpms compatibility with target kernel...
This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do not reinstall them. Do you want to continue?[y/N]:y rpm --nosignature -e --allmatches --nodeps rdma rdma Starting mlnx-en-4.4-2.0.7.0 installation ... Installing mlnx-en-utils 4.4 RPM
Preparing... ##################################################
mlnx-en-utils ##################################################
Installing kmod-mlnx-en 4.4 RPM
Preparing... ##################################################
kmod-mlnx-en ##################################################
Installing mlnx-en-sources 4.4 RPM
Preparing... ##################################################
mlnx-en-sources ##################################################
Installing mlnx-en-doc 4.4 RPM
Preparing... ##################################################
mlnx-en-doc ##################################################
Installing user level RPMs:
Preparing... ##################################################
ofed-scripts ##################################################
Preparing... ##################################################
mstflint ##################################################
Device (83:00.0):
83:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Link Width: x8
PCI Link Speed: 8GT/s Device (83:00.1):
83:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Link Width: x8
PCI Link Speed: 8GT/s Installation finished successfully. Preparing... ########################################### [100%]
1:mlnx-fw-updater ########################################### [100%]
Updated /usr/share/hwdata/pci.ids
Attempting to perform Firmware update...
Querying Mellanox devices firmware ... Device #1:
---------- Device Type: ConnectX4LX
Part Number: MCX4121A-XCA_Ax
Description: ConnectX-4 Lx EN network interface card; 10GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
PSID: MT_2420110004
PCI Device Name: 83:00.0
Base MAC: ec0d9ad2fd68
Versions: Current Available
FW 14.20.1010 14.23.1020
PXE 3.5.0210 3.5.0504
UEFI N/A 14.16.0017 Status: Update required ---------
Found 1 device(s) requiring firmware update... Device #1: Updating FW ... Done Restart needed for updates to take effect.
Log File: /tmp/mlnx-en.28728.logs/fw_update.log
Configuring /etc/security/limits.conf.
To load the new driver, run:
/etc/init.d/mlnx-en.d restart
4，重启服务：
/etc/init.d/mlnx-en.d restart
5，安装MLNX_OFED_LINUX-4.4
这里不用像软件RDMA 那样的启动rxe_cfg了。
yum -y install libmml tcl tk libmnl
tar fx MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64.tgz
cd MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64
./mlnxofedinstall --add-kernel-support --skip-repo
/etc/init.d/openibd restart # 这个命令最好在管理卡上执行，xshell上执行有可能导致网卡掉IP，
/etc/init.d/network restart
chkconfig openibd on
ibv_devices
出现一下结果代表成功：
# ibv_devices
device node GUID
------ ----------------
mlx5_1 ec0d9a0300d2fc99
mlx5_0 ec0d9a0300d2fc98 如果这一步不成功（有时候rxe_cfg不启动也可以）：
# rxe_cfg start (并绑定eth4网卡)
# ibv_devices
device node GUID
------ ----------------
rxe0 ee0d9afffed2fd68
# ibv_devinfo rxe0
hca_id: rxe0
transport: InfiniBand (0)
fw_ver: 0.0.0
node_guid: ee0d:9aff:fed2:fd68
sys_image_guid: 0000:0000:0000:0000
vendor_id: 0x0000
vendor_part_id: 0
hw_ver: 0x0
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
6，利用rping 命令测试：
生成server端：
[root@bj01-prd-hadoop499.vivo.lan:/root]
# rping -s -a 10.20.15.23 -v -C 10
生成client端：
client端的安装和服务端一样，生成命令是：
# rping -c -a 10.20.15.23 -v -C 10
此时就会出现一下界面，证明安装成功：
# rping -s -a 10.20.15.23 -v -C 10
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
# rping -c -a 10.20.15.23 -v -C 10
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
client DISCONNECT EVENT...
------------------------------------------------------
利用udaddy来测试，出现一下结果代表成功：
服务端：
[root@bj01-prd-hadoop499.vivo.lan:/mnt/MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64]
# udaddy
udaddy: starting server
receiving data transfers
sending replies
data transfers complete
test complete
return status 0
客户端：
[root@bj01-prd-hadoop500.vivo.lan:/root]
# udaddy -s 10.20.15.23
udaddy: starting client
udaddy: connecting
initiating data transfers
receiving data transfers
data transfers complete
test complete
return status 0