问题来源：使用了tf的fft系列函数
问题显示：
2021-11-29 16:48:33.400178: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-11-29 16:48:38.176008: E tensorflow/stream_executor/cuda/cuda_fft.cc:223] failed to make cuFFT batched plan:5
2021-11-29 16:48:38.176052: E tensorflow/stream_executor/cuda/cuda_fft.cc:426] Initialize Params: rank: 1 elem_count: 2048 input_embed: 2048 input_stride: 1 input_distance: 2048 output_embed: 1025 output_stride: 1 output_distance: 1025 batch_count: 480
解决方案：
在代码中设置允许显存增长
from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)
Reference