一. 压缩文件和解压文件
1. 压缩文件
tar -czvf UNITE-main.tar.gz ./UNITE-main/
2. 解压文件
tar -xvf ./UNITE-main/
二. CUDA更改为NPU
import torch_npu
data['label'] = data['label'].cuda()
data['instance'] = data['instance'].cuda()
data['image'] = data['image'].cuda()
更改为
data['label'] = data['label'].npu()
data['instance'] = data['instance'].npu()
data['image'] = data['image'].npu()
三. 配置环境变量
1. 创建env.sh
touch env.sh
2. 打开env.sh
vi env.sh
3. 配置环境变量
# 配置CANN相关环境变量
CANN_INSTALL_PATH_CONF='/etc/Ascend/ascend_cann_install.info'
if [ -f $CANN_INSTALL_PATH_CONF ]; then
DEFAULT_CANN_INSTALL_PATH=$(cat $CANN_INSTALL_PATH_CONF | grep Install_Path | cut -d "=" -f 2)
else
DEFAULT_CANN_INSTALL_PATH="/usr/local/Ascend/"
fi
CANN_INSTALL_PATH=${
1:-${
DEFAULT_CANN_INSTALL_PATH}}
if [ -d ${
CANN_INSTALL_PATH}/ascend-toolkit/latest ];then
source ${
CANN_INSTALL_PATH}/ascend-toolkit/set_env.sh
else
source ${
CANN_INSTALL_PATH}/nnae/set_env.sh
fi
# 导入依赖库
export LD_LIBRARY_PATH=${
LD_LIBRARY_PATH}:/usr/local/openblas/lib
export LD_LIBRARY_PATH=${
LD_LIBRARY_PATH}:/usr/local/lib/
export LD_LIBRARY_PATH=${
LD_LIBRARY_PATH}:/usr/lib64/
export LD_LIBRARY_PATH=${
LD_LIBRARY_PATH}:/usr/lib/
export LD_LIBRARY_PATH=${
LD_LIBRARY_PATH}:/usr/lib/aarch64_64-linux-gnu
# 配置自定义环境变量
export HCCL_WHITELIST_DISABLE=1
# log
export ASCEND_SLOG_PRINT_TO_STDOUT=0 # 日志打屏, 可选
export ASCEND_GLOBAL_LOG_LEVEL=3 # 日志级别常用 1 INFO级别; 3 ERROR级别
export ASCEND_GLOBAL_EVENT_ENABLE=0 # 默认不使能event日志信息
并输入
:wq!
4. 使用环境
source env.sh
四. RuntimeError: ACL stream synchronize failed, error code:507018
E39999: Inner Error, Please contact support engineer!
E39999 Aicpu kernel execute failed, device_id=0, stream_id=0, task_id=6394, fault op_name=ScatterElements[FUNC:GetError][FILE:stream.cc][LINE:1044]
TraceBack (most recent call last):
rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[41411]:
EXCEPTION TASK:
Exception info:TGID=2593324, model id=65535, stream id=0, stream phase=SCHEDULE, task id=742, task type=aicpu kernel, recently received task id=742, recently send task id=741, task phase=RUN
Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
Other info[0]:time=2023-10-12-11:22:01.273.951, function=proc_aicpu_task_done, line=972, error code=0x2a
EXCEPTION TASK:
Exception info:TGID=2593324, model id=65535, stream id=0, stream phase=3, task id=6394, task type=aicpu kernel, recently received task id=6406, recently send task id=6393, task phase=RUN
Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
Other info[0]:time=2023-10-12-11:41:20.661.958, function=proc_aicpu_task_done, line=972, error code=0x2a
Traceback (most recent call last):
File "train.py", line 40, in <module>
trainer.run_generator_one_step(data_i)
File "/home/ma-user/work/SPADE-master/trainers/pix2pix_trainer.py", line 35, in run_generator_one_step
g_losses, generated = self.pix2pix_model(data, mode='generator')
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs, **kwargs)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ma-user/work/SPADE-master/models/pix2pix_model.py", line 43, in forward
input_semantics, real_image = self.preprocess_input(data)
File "/home/ma-user/work/SPADE-master/models/pix2pix_model.py", line 113, in preprocess_input
data['label'] = data['label'].npu()
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch_npu/utils/device_guard.py", line 38, in wrapper
return func(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch_npu/utils/tensor_methods.py", line 66, in _npu
return torch_npu._C.npu(self, *args, **kwargs)
RuntimeError: ACL stream synchronize failed, error code:507018
THPModule_npu_shutdown success.
猜测可能是没有开混合精度
五. 开启混合精度
1. 在构建神经网络前，我们需要导入torch_npu中的AMP模块
import time
import torch
import torch.nn as nn
import torch_npu
from torch_npu.npu import amp # 导入AMP模块
2. 在模型、优化器定义之后，定义AMP功能中的GradScaler
model = CNN().to(device)
train_dataloader = DataLoader(train_data, batch_size=batch_size) # 定义DataLoader
loss_func = nn.CrossEntropyLoss().to(device) # 定义损失函数
optimizer = torch.optim.SGD(model.parameters(), lr=0.1) # 定义优化器
scaler = amp.GradScaler() # 在模型、优化器定义之后，定义GradScaler
3. 在训练代码中添加AMP功能相关的代码开启AMP
for epo in range(epochs):
for imgs, labels in train_dataloader:
imgs = imgs.to(device)
labels = labels.to(device)
with amp.autocast():
outputs = model(imgs) # 前向计算
loss = loss_func(outputs, labels) # 损失函数计算
optimizer.zero_grad()
# 进行反向传播前后的loss缩放、参数更新
scaler.scale(loss).backward() # loss缩放并反向传播
scaler.step(optimizer) # 更新参数（自动unscaling）
scaler.update() # 基于动态Loss Scale更新loss_scaling系数
六. 未知错误
E39999: Inner Error, Please contact support engineer!
E39999 An exception occurred during AICPU execution, stream_id:78, task_id:742, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:673]
TraceBack (most recent call last):
Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:task.cc][LINE:1068]
Aicpu kernel execute failed, device_id=0, stream_id=78, task_id=742.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:774]
Aicpu kernel execute failed, device_id=0, stream_id=78, task_id=742, fault op_name=ScatterElements[FUNC:GetError][FILE:stream.cc][LINE:1044]
rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]
op[Minimum], The Minimum op dtype is not same, type1:DT_FLOAT16, type2:DT_FLOAT[FUNC:CheckTwoInputDtypeSame][FILE:util.cc][LINE:116]
Verifying Minimum failed.[FUNC:InferShapeAndType][FILE:infershape_pass.cc][LINE:135]
Call InferShapeAndType for node:Minimum(Minimum) failed[FUNC:Infer][FILE:infershape_pass.cc][LINE:117]
process pass InferShapePass on node:Minimum failed, ret:4294967295[FUNC:RunPassesOnNode][FILE:base_pass.cc][LINE:530]
build graph failed, graph id:894, ret:1343242270[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1484]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 1343242270[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
DEVICE[0] PID[189368]:
EXCEPTION TASK:
Exception info:TGID=3114744, model id=65535, stream id=78, stream phase=SCHEDULE, task id=742, task type=aicpu kernel, recently received task id=742, recently send task id=741, task phase=RUN
Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
Other info[0]:time=2023-10-12-12:12:22.763.259, function=proc_aicpu_task_done, line=972, error code=0x2a
EXCEPTION TASK:
Exception info:TGID=3114744, model id=65535, stream id=78, stream phase=3, task id=4347, task type=aicpu kernel, recently received task id=4354, recently send task id=4346, task phase=RUN
Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
Other info[0]:time=2023-10-12-12:13:57.997.757, function=proc_aicpu_task_done, line=972, error code=0x2a
Aborted (core dumped)
(py38) [ma-user SPADE-master]$Process ForkServerProcess-2:
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 61, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 58, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 268, in task_distribute
key, func_name, detail = resource_proxy[TASK_QUEUE].get()
File "<string>", line 2, in get
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/multiprocessing/managers.py", line 835, in _callmethod
kind, result = conn.recv()
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError
/home/ma-user/anaconda3/envs/py38/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 91 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
七. 数据集加载错误
Traceback (most recent call last):
File "train.py", line 126, in <module>
content_images = next(content_iter).to(device)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
data = self._next_data()
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 569, in _next_data
index = self._next_index() # may raise StopIteration
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in _next_index
return next(self._sampler_iter) # may raise StopIteration
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 226, in __iter__
for idx in self.sampler:
File "/home/ma-user/work/CCPL-main/sampler.py", line 10, in InfiniteSampler
yield order[i]
IndexError: index -1 is out of bounds for axis 0 with size 0
THPModule_npu_shutdown success.
数据集加载错误，需要改成
../iRay/train/nightA
八. 梯度错误
Traceback (most recent call last):
File "train.py", line 129, in <module>
loss_c, loss_s, loss_ccp = network(content_images, style_images, args.tau, args.num_s, args.num_l)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ma-user/work/CCPL-main/net.py", line 280, in forward
loss_s = self.calc_style_loss(g_t_feats[0], style_feats[0])
File "/home/ma-user/work/CCPL-main/net.py", line 265, in calc_style_loss
input_mean, input_std = calc_mean_std(input)
File "/home/ma-user/work/CCPL-main/function.py", line 8, in calc_mean_std
feat_var = feat.view(N, C, -1).var(dim=2) + eps
RuntimeError: cannot resize variables that require grad
错误提示 “RuntimeError: cannot resize variables that require grad” 指示无法调整需要梯度计算的变量的大小。这通常是由于尝试在具有梯度的张量上执行不可微分操作（如 .view()）导致的。
修改为
feat_var = feat.data.view(N, C, -1).var(dim=2) + eps
九. Tensor类型错误
Traceback (most recent call last):
File "train.py", line 129, in <module>
loss_c, loss_s, loss_ccp = network(content_images, style_images, args.tau, args.num_s, args.num_l)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ma-user/work/CCPL-main/net.py", line 287, in forward
loss_ccp = self.CCPL(g_t_feats, content_feats, num_s, start_layer, end_layer)
File "/home/ma-user/anaconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ma-user/work/CCPL-main/net.py", line 211, in forward
f_q, sample_ids = self.NeighborSample(feats_q[i], i, num_s, [])
File "/home/ma-user/work/CCPL-main/net.py", line 179, in NeighborSample
print(feat_r[:, c_ids, :].shape, feat_r[:, n_ids, :].shape)
IndexError: tensors used as indices must be long, byte or bool tensors
错误提示指出索引操作中使用的张量必须是 long、byte 或 bool 类型的张量。请确保 c_ids 和 n_ids 是正确的张量类型。如果它们是其他类型的张量，如 float 或 int 类型，请将它们转换为 long 类型。可以使用 .long() 方法将张量转换为 long 类型。
c_ids = c_ids.long()
n_ids = n_ids.long()
十. Tensor位置错误
Traceback (most recent call last):
File "train.py", line 59, in <module>
visualizer.display_current_results(visuals, epoch, iter_counter.total_steps_so_far)
File "/home/ma-user/work/SPADE-master/util/visualizer.py", line 45, in display_current_results
visuals = self.convert_visuals_to_numpy(visuals)
File "/home/ma-user/work/SPADE-master/util/visualizer.py", line 134, in convert_visuals_to_numpy
t = util.tensor2im(t, tile=tile)
File "/home/ma-user/work/SPADE-master/util/util.py", line 77, in tensor2im
one_image_np = tensor2im(one_image)
File "/home/ma-user/work/SPADE-master/util/util.py", line 88, in tensor2im
image_numpy = image_tensor.detach().npu().float().numpy()
TypeError: can't convert xla:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
THPModule_npu_shutdown success.
这个错误提示表明在将张量转换为NumPy数组时出现了问题。错误信息中提到了不能将类型为xla:0的设备类型张量转换为NumPy数组。它建议你先使用Tensor.cpu()将张量复制到主机内存中。
对于解决这个问题，你可以修改代码中的相关部分，确保在将张量转换为NumPy数组之前将其从设备中移动到主机内存中。具体而言，你可以在调用tensor2im()函数之前添加Tensor.cpu()方法。
image_numpy = image_tensor.detach().cpu().float().numpy()
通过这种方式，你将在将张量转换为NumPy数组之前将其移动到CPU上，并且不再会出现无法将xla:0设备类型张量转换为NumPy数组的错误。
十一. 数据集加载错误
Traceback (most recent call last):
File "main.py", line 51, in <module>
main(config)
File "main.py", line 33, in main
solver = Solver(config, get_loader(config))
File "/home/ma-user/work/SCFT_with_gradient_cosine_logging/data_loader.py", line 107, in get_loader
drop_last=True)
File "/home/ma-user/anaconda3/envs/PyTorch-1.11/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 277, in __init__
sampler = RandomSampler(dataset, generator=generator) # type: ignore[arg-type]
File "/home/ma-user/anaconda3/envs/PyTorch-1.11/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 98, in __init__
"value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0
**
原因：shuffle的参数设置错误导致。
解决方法：因为已经有batch_size了，就不需要shuffle来进行随机了，将shuffle设置为FALSE即可。（网上这么说的，更改之后确实可以用）**
十二. python错误
main.py:49: ResourceWarning: unclosed file <_io.TextIOWrapper name='config.yml' mode='r' encoding='UTF-8'>
config = yaml.load(open(params.config, 'r'), Loader=yaml.FullLoader)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
原因分析： 缺少close（）
def get_data(file_name):
rows = []
testReportDir = "../test/"
testReportDir_FileName = testReportDir + file_name
data_file = open(testReportDir_FileName, mode="r", encoding="utf-8")
reader = csv.reader(data_file)
next(reader, None)
for row in reader:
rows.append(row)
return rows
with open(testReportDir_FileName, mode="r", encoding="utf-8") as f:
data_file = f.read()
with open的用途：是python用来打开本地文件的，它会在使用完毕后，自动关闭文件，无需手动书写close()（网上这么说，没影响，未改）