问题描述：

按照示例教程提供的脚本[./script/run_distribute_train.sh]，进行多卡单模型训练时，出现了以下报错：

请问有人遇到过这种错误吗？能否分享一下解决方法？谢谢！

[CRITICAL] DISTRIBUTED(21714,7fa32ebc0740,python):2022-03-09-08:30:23.606.377 [mindspore/ccsrc/distributed/cluster/cluster_context.cc:164] InitNodeRole] Role name '' is invalid. Maybe you are trying to call 'mindspore.communication.init()' without using 'mpirun', which will make MindSpore load several environment variables and check their validation. Please use 'mpirun' to launch this process to fix this issue, or refer to this link if you want to run distributed training without using 'mpirun': https://www.mindspore.cn/docs/programming_guide/zh-CN/master/distributed_training_gpu.html#openmpi.

{'data_dir': 'Train dataset directory.', 'per_batch_size': 'Batch size for Training.', 'pretrained_backbone': 'The ckpt file of CspDarkNet53.', 'resume_yolov4': 'The ckpt file of YOLOv4, which used to fine tune.', 'pretrained_checkpoint': 'The ckpt file of YoloV4CspDarkNet53.', 'filter_weight': 'Filter the last weight parameters', 'transfer_train': 'If set it True, transfer training on other dataset, and set filter_weight True', 'lr_scheduler': 'Learning rate scheduler, options: exponential, cosine_annealing.', 'lr': 'Learning rate.', 'lr_epochs': "Epoch of changing of lr changing, split with ','.", 'lr_gamma': 'Decrease lr by a factor of exponential lr_scheduler.', 'eta_min': 'Eta_min in cosine_annealing scheduler.', 't_max': 'T-max in cosine_annealing scheduler.', 'max_epoch': 'Max epoch num to train the model.', 'warmup_epochs': 'Warmup epochs.', 'weight_decay': 'Weight decay factor.', 'momentum': 'Momentum.', 'loss_scale': 'Static loss scale.', 'label_smooth': 'Whether to use label smooth in CE.', 'label_smooth_factor': 'Smooth strength of original one-hot.', 'log_interval': 'Logging interval steps.', 'ckpt_path': 'Checkpoint save location.', 'ckpt_interval': 'Save checkpoint interval.', 'is_save_on_master': 'Save ckpt on master or all rank, 1 for master, 0 for all ranks.', 'is_distributed': 'Distribute train or not, 1 for yes, 0 for no.', 'rank': 'Local rank of distributed.', 'group_size': 'World size of device.', 'need_profiler': 'Whether use profiler. 0 for no, 1 for yes.', 'training_shape': 'Fix training shape.', 'resize_rate': 'Resize rate for multi-scale training.', 'run_eval': 'Run evaluation when training.', 'save_best_ckpt': 'Save best checkpoint when run_eval is True.', 'eval_start_epoch': 'Evaluation start epoch when run_eval is True.', 'eval_interval': 'Evaluation interval when run_eval is True', 'ann_file': 'path to annotation', 'each_multiscale': 'Apply multi-scale for each scale', 'detect_head_loss_coff': 'the loss coefficient of detect head. The order of coefficients is large head, medium head and small head', 'bbox_class_loss_coff': 'bbox and class loss coefficient. The order of coefficients is ciou loss, confidence loss and class loss', 'labels': 'the label of train data', 'mosaic': 'use mosaic data augment', 'multi_label': 'use multi label to nms', 'multi_label_thresh': 'multi label thresh', 'pretrained': 'model_path, local pretrained model to load', 'log_path': 'checkpoint save location', 'ann_val_file': 'path to annotation', 'device_id': 'Device id for export', 'batch_size': 'batch size for export', 'testing_shape': 'shape for test', 'ckpt_file': 'Checkpoint file path for export', 'file_name': 'output file name for export', 'file_format': 'file format for export', 'keep_detect': 'keep the detect module or not, default: True', 'img_id_file_path': 'path of image dataset', 'result_files': 'path to 310 infer result floder'}

Traceback (most recent call last):

  File "train.py", line 303, in <module>

    run_train()

  File "/dev/shm/models-master/official/cv/yolo4/scripts/train_parallel5/model_utils/moxing_adapter.py", line 105, in wrapped_func

    run_func(*args, **kwargs)

  File "train.py", line 181, in run_train

    profiler = set_default()

  File "train.py", line 68, in set_default

    init("nccl")

  File "/root/anaconda3/lib/python3.7/site-packages/mindspore/communication/management.py", line 146, in init

    init_gpu_collective()

RuntimeError: mindspore/ccsrc/distributed/cluster/cluster_context.cc:164 InitNodeRole] Role name '' is invalid. Maybe you are trying to call 'mindspore.communication.init()' without using 'mpirun', which will make MindSpore load several environment variables and check their validation. Please use 'mpirun' to launch this process to fix this issue, or refer to this link if you want to run distributed training without using 'mpirun': https://www.mindspore.cn/docs/programming_guide/zh-CN/master/distributed_training_gpu.html#openmpi.

解答：

1. GPU 环境上面不能用for循环的方式启动。

2. 指定4，7号卡跑，需要一个slots=2的 host files，例如 server_ip slots=2

然后用下面的方式启动

export CUDA_VISIBLE_DEVICES=4,7

mpirun xxxx

1. 这种报错给出了3种可能，一般是因为报错中给出的原因2和3，请检查下