解决PyTorch报错“cublas runtime error : library not initialized”

时间：2025-05-06 08:37:52

运行一段深度学习的training代码时遇到如下错误：

  File "train_mdnet.py", line 69, in train_mdnet
    neg_regions = neg_regions.cuda()
  File "/home/dodge/anaconda3/envs/py27_for_pytorch020/lib/python2.7/site-packages/torch/autograd/", line 279, in cuda
    return (self, device_id, async)
  File "/home/dodge/anaconda3/envs/py27_for_pytorch020/lib/python2.7/site-packages/torch/autograd/_functions/", line 151, in forward
    return (async=async)
  File "/home/dodge/anaconda3/envs/py27_for_pytorch020/lib/python2.7/site-packages/torch/_utils.py", line 66, in _cuda
    return new_type(()).copy_(self, async)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1511304568725/work/torch/lib/THC/generic/:66

经查阅PyTorch社区，发现该问题主要可能由两方面的原因引起：

  1. GPU显存不足
  2. GPU缓存没有清理干净

检查主机实际运行情况，发现还有一个CUDA程序处于debug状态，应该是显存不够了所致，停止该程序，问题解决。

相关文章