1、朴素All Reduce
假设两两GPU间的网络带宽为1GB/s,梯度总量1GB,
参考链接
1)Baidu All Reduce:
https://github.com/baidu-research/baidu-allreduce
2)Horovod:
1、朴素All Reduce
假设两两GPU间的网络带宽为1GB/s,梯度总量1GB,
参考链接
1)Baidu All Reduce:
https://github.com/baidu-research/baidu-allreduce
2)Horovod: