工作环境
系统：Ubuntu LTS 显卡：GPU NVIDIA驱动：410.93 CUDA：10.0 Python：.x
CUDA以及NVIDIA驱动安装，详见https://www.cnblogs.com/orzs/p/10951473.html
需要部署的软件
conda环境 nccl2环境 openmpi环境 horovod环境
1. 创建conda环境
官网下载地址：https://www.anaconda.com/distribution/#download-section
下载合适的安装文件，然后运行。
cd init sudo wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh bash Anaconda3-2019.03-Linux-x86_64.sh
根据提示操作，并选择安装目录，默认安装在~/anaconda3/ 目录下。
注：初始化操作
1、如果默认不初始化，则安装之后，没有conda命令，需要手动初始化
注：为避免用户名泄露，此处的用户名均已$USER替代
installation finished.
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
[no] >>>
You have chosen to not have conda modify your shell scripts at all.
To activate conda's base environment in your current shell session:
eval "$(/home/$USER/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)"
To install conda's shell functions for easier access, first activate, then:
conda init
If you'd prefer that conda's base environment not be activated on startup,
set the auto_activate_base parameter to false:
conda config --set auto_activate_base false
Thank you for installing Anaconda3!
===========================================================================
Anaconda and JetBrains are working together to bring you Anaconda-powered
environments tightly integrated in the PyCharm IDE.
PyCharm for Anaconda is available at:
https://www.anaconda.com/pycharm
2、如果选择初始化，则会修改~/.bashrc文件，并创建conda命令 installation finished. Do you wish the installer to initialize Anaconda3 by running conda init? [yes|no] "deeplearning" 105L, 3558C written installation finished. Do you wish the installer to initialize Anaconda3 by running conda init? [yes|no] [no] >>> yes WARNING: The conda.compat module is deprecated and will be removed in a future release. no change /home/$USER/anaconda3/condabin/conda no change /home/$USER/anaconda3/bin/conda no change /home/$USER/anaconda3/bin/conda-env no change /home/$USER/anaconda3/bin/activate no change /home/$USER/anaconda3/bin/deactivate no change /home/$USER/anaconda3/etc/profile.d/conda.sh no change /home/$USER/anaconda3/etc/fish/conf.d/conda.fish no change /home/$USER/anaconda3/shell/condabin/Conda.psm1 no change /home/$USER/anaconda3/shell/condabin/conda-hook.ps1 no change /home/$USER/anaconda3/lib/python3./site-packages/xonsh/conda.xsh no change /home/$USER/anaconda3/etc/profile.d/conda.csh modified /home/$USER/.bashrc ==> For changes to take effect, close and re-open your current shell. <== If you'd prefer that conda's base environment not be activated on startup, set the auto_activate_base parameter to false: conda config --set auto_activate_base false Thank you for installing Anaconda3! =========================================================================== Anaconda and JetBrains are working together to bring you Anaconda-powered environments tightly integrated in the PyCharm IDE. PyCharm for Anaconda is available at: https://www.anaconda.com/pycharm
执行以下命令，使conda环境生效
source ~/.bashrc
2. 进入conda py3.6
conda create -n py36 python=3.6 conda activate py36
3. 安装必要包
#修改清华的pip源
mkdir ~/.pip touch ~/.pip/pip.conf
#pip.conf中写入以下内容
[global] index-url = https://pypi.tuna.tsinghua.edu.cn/simple
安装包
pip pip install opencv-python==4.1.0.25 pip pip
4. 安装nccl2
下载地址：https://docs.nvidia.com/deeplearning/sdk/nccl-install-guide/index.html
根据系统和cuda版本下载对应的nccl2
-ga-cuda10.0_1-1_amd64.deb -ga-cuda10./7fa2af80.pub（根据提示执行） sudo apt update -+cuda10. libnccl-dev=-+cuda10.
5、安装libcudnn
根据版本，下载对应的文件：https://developer.nvidia.com/rdp/cudnn-download
-+cuda10.0_amd64.deb -+cuda10.0_amd64.deb
6. 安装openmpi
下载地址：https://www.open-mpi.org/faq/?category=building#easy-build
sudo wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.gz .tar.gz | tar xf - cd openmpi-/ sudo ./configure --prefix=/usr/local sudo make all install
7. 安装horovod
文档说明：https://github.com/horovod/horovod/blob/master/docs/gpus.rst
HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir horovod
注：HOROVOD_WITH_TENSORFLOW=1 可开启debug模式。
至此，深度学习环境安装完成，接下来即可做深度训练。
conda环境常用命令
如何默认不使用conda环境
conda config --set auto_activate_base false
退出conda环境
conda deactivate
进入conda环境
conda activate
安装过程中可能出现的问题：
1、
ImportError: libcudnn.so.: cannot open shared object file: No such file or directory
原因：cudann未安装或者版本错误
解决：根据版本，下载对应的文件：https://developer.nvidia.com/rdp/cudnn-download
-+cuda10.0_amd64.deb -+cuda10.0_amd64.deb
2、
ImportError: libcuda.so.: cannot open shared object file: No such file or directory
原因：一般是cuda版本不对导致
解决：安装对应的cuda版本即可
3、
ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
原因：一般情况是cuda链接库的问题
解决：执行以下命令即可
sudo ldconfig /usr/local/cuda/lib64
4、奇葩问题：
ModuleNotFoundError: No module named 'cv2'
如果未安装opencv-python，直接执行以下命令安装即可
pip install opencv-python==4.1.0.25
如果已经安装，依然错误提示，我遇到的情况是，Python被劫持
执行命令
which python
回显提示
~/anaconda3/envs/py36/bin/python
执行
~/anaconda3/envs/py36/bin/python
看到的版本是3.6.8
但是直接python看到是3.6.6
原因：python被劫持
解决：将~/.bashrc里的python环境变量清除即可
# alias python=/usr/bin/python3.
5、执行以下命令报错
conda create -n py36 python=3.6
WARNING: The conda.compat module is deprecated and will be removed in a future release. Collecting package metadata: failed UnavailableInvalidChannel: The channel is not accessible or is invalid. channel name: anaconda/pkgs/free channel url: https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free error code: You will need to adjust your conda configuration to proceed. Use `conda config --show channels` to view your configuration's current state, and use `conda config --show-sources` to view config file locations.
检查conda配置（以前曾经安装过conda）
conda config --show-sources
channels: - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/ - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ - defaults show_channel_urls: True
原因：conda已经不支持外部源
解决：删除清华的源即可
conda config --remove channels 'https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/' conda config --remove channels 'https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/'
6、
[$USER-nmg-:] mca_base_component_repository_open: unable to open mca_oob_ud: libibverbs.so.: cannot open shared object file: No such file or directory (ignored) [$USER-nmg-:] mca_base_component_repository_open: unable to open mca_oob_ud: libibverbs.so.: cannot open shared object file: No such file or directory (ignored) [$USER-nmg-:] mca_base_component_repository_open: unable to open mca_btl_openib: libibverbs.so.: cannot open shared object file: No such file or directory (ignored)
原因：缺少libibverbs.so.1导致
解决：安装libibverbs1即可
apt-cache search libibverbs sudo apt-get install libibverbs1
7、
python: symbol lookup error: /usr/local/lib/openmpi/mca_coll_cuda.so: undefined symbol: opal_cuda_check_bufs
原因：openmpi安装有问题或者版本冲突导致
解决：卸载并重新安装openmpi即可。
cd /where/your/old_mpi/sources/are //进入其他版本的安装目录 sudo make uninstall sudo rm -rf /usr/local/lib/openmpi /usr/local/lib/libmca* /usr/local/lib/libmpi* /usr/local/lib/libompitrace* /usr/local/lib/libopen* /usr/local/lib/liboshmem* /usr/local/lib/mpi_* cd /where/your/mpi/sources/are //进入需要安装的版本的目录 sudo ./configure --prefix=/usr/local sudo make all install
8、
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
我遇到的情况是，已经安装了对应的版本（cuda10.0、libcudnn7-dev_7.6.0.64、tensorflow-gpu-1.13.1），但是被/usr/local/cuda-9.0/空目录影响到了，删除此目录即可。
sudo rm -rf /usr/local/cuda-9.0/