1、Loaded runtime CuDNN library: 7103 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.
解析:cuda与cudnn版本不匹配问题
conda install cudatoolkit=9.0
conda install cudnn=7.1.2
安装参考:http://www.sohu.com/a/225953058_491081
2、InvalidArgumentError (see above for traceback): Integer division by zero
[[Node: training/SGD/gradients/mrcnn_bbox_loss_1/concat_grad/mod = FloorMod[T=DT_INT32, _class=["loc:@mrcnn_bbox_loss_1/concat"], _device="/job:localhost/replica:0/task:0/cpu:0"](mrcnn_bbox_loss_1/concat/axis, training/SGD/gradients/mrcnn_bbox_loss_1/concat_grad/Rank)]]
解析:更新tensorflow-gpu版本到1.7
3、 Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.13GiB. The caller indicates that this 。。。
解决方法:方法一:batch_size 小一点;
方法二:(没有使用过)
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config = config)
参考:https://stackoverflow.com/questions/36927607/how-can-i-solve-ran-out-of-gpu-memory-in-tensorflow