Runtime Error in baseline code.

[yihong-chen]

Hi, Thanks for setting up the discussion forum.

I came across the following issue

RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

when I tried to reproduce the baseline using

dglke_train --model_name TransE_l2 \
--hidden_dim 200 --gamma 10 --lr 0.1 --regularization_coef 1e-9 \
--valid --test -adv \
--mix_cpu_gpu --num_proc 2 --num_thread 2 --gpu 0 1 --async_update --force_sync_interval 10000 --no_save_emb --print_on_screen \
--encoder_model_name roberta --save_path $SAVE_PATH \
--data_path ./data/

I also tried using 'spawn' as the error suggests but it still doesn't work. I wonder if there is any other workaround for this issue.

Environments

• torch 1.8.0
• ogb 1.3.0
• dgl 0.4.3.post2
• cuda 10.2
• python 3.7.10

The entire output looks like this

Using backend: pytorch
To build training dataset
|Train|: 501160482
random partition 501160482 edges into 2 parts
part 0 has 250580241 edges
part 1 has 250580241 edges
/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will treat all graphs as multigraph in the future.
  warnings.warn(msg, warn_type)
Training dataset built, it takes 247 seconds
Building training sampler for proc 0
Training sampler for proc 0 created, it takes 42.49349904060364 seconds
Building training sampler for proc 1
Training sampler for proc 1 created, it takes 41.20206379890442 seconds
To create eval_dataset
eval_dataset created, it takes 5 seconds
creating valid sampler for proc 0
0 850292
Valid sampler for proc 0 created, it takes 0.0001957416534423828 seconds
creating valid sampler for proc 1
850292 1700584
Valid sampler for proc 1 created, it takes 2.6464462280273438e-05 seconds
creating test sampler for proc 0
0 679651
Test sampler for proc 0 created, it takes 0.0007002353668212891 seconds
creating test sampler for proc 1
679651 1359303
Test sampler for proc 1 created, it takes 0.0006911754608154297 seconds
To create model
False False
Model created, it takes 0.014177083969116211 seconds
Total initialize time 341.349 seconds
2021-03-23 14:33:58,578 INFO     model_name          :TransE_l2
2021-03-23 14:33:58,579 INFO     data_path           :./data/
2021-03-23 14:33:58,579 INFO     dataset             :wikikg90m
2021-03-23 14:33:58,580 INFO     format              :built_in
2021-03-23 14:33:58,580 INFO     data_files          :None
2021-03-23 14:33:58,580 INFO     delimiter           :
2021-03-23 14:33:58,580 INFO     save_path           :./experiments/exp_1/run_baseline/TransE_l2_wikikg90m_roberta_d_200_g_10.09
2021-03-23 14:33:58,580 INFO     no_save_emb         :True
2021-03-23 14:33:58,580 INFO     max_step            :1000000
2021-03-23 14:33:58,580 INFO     batch_size          :400
2021-03-23 14:33:58,580 INFO     batch_size_eval     :50
2021-03-23 14:33:58,581 INFO     neg_sample_size     :100
2021-03-23 14:33:58,581 INFO     neg_deg_sample      :False
2021-03-23 14:33:58,581 INFO     neg_deg_sample_eval :False
2021-03-23 14:33:58,581 INFO     neg_sample_size_eval:1000
2021-03-23 14:33:58,581 INFO     eval_percent        :1
2021-03-23 14:33:58,581 INFO     no_eval_filter      :False
2021-03-23 14:33:58,581 INFO     log_interval        :1000
2021-03-23 14:33:58,581 INFO     eval_interval       :50000
2021-03-23 14:33:58,581 INFO     test                :True
2021-03-23 14:33:58,581 INFO     num_proc            :2
2021-03-23 14:33:58,582 INFO     num_thread          :2
2021-03-23 14:33:58,582 INFO     force_sync_interval :1000
2021-03-23 14:33:58,582 INFO     hidden_dim          :200
2021-03-23 14:33:58,582 INFO     lr                  :0.1
2021-03-23 14:33:58,582 INFO     gamma               :10.0
2021-03-23 14:33:58,582 INFO     double_ent          :False
2021-03-23 14:33:58,582 INFO     double_rel          :False
2021-03-23 14:33:58,582 INFO     neg_adversarial_sampling:True
2021-03-23 14:33:58,582 INFO     adversarial_temperature:1.0
2021-03-23 14:33:58,583 INFO     regularization_coef :1e-09
2021-03-23 14:33:58,583 INFO     regularization_norm :3
2021-03-23 14:33:58,583 INFO     pairwise            :False
2021-03-23 14:33:58,583 INFO     loss_genre          :Logsigmoid
2021-03-23 14:33:58,583 INFO     margin              :1.0
2021-03-23 14:33:58,583 INFO     gpu                 :[0, 1]
2021-03-23 14:33:58,583 INFO     mix_cpu_gpu         :True
2021-03-23 14:33:58,583 INFO     valid               :True
2021-03-23 14:33:58,583 INFO     rel_part            :False
2021-03-23 14:33:58,584 INFO     async_update        :True
2021-03-23 14:33:58,584 INFO     has_edge_importance :False
2021-03-23 14:33:58,584 INFO     print_on_screen     :True
2021-03-23 14:33:58,584 INFO     encoder_model_name  :roberta
2021-03-23 14:33:58,584 INFO     mlp_lr              :0.0001
2021-03-23 14:33:58,584 INFO     seed                :0
2021-03-23 14:33:58,584 INFO     eval_filter         :True
2021-03-23 14:33:58,584 INFO     soft_rel_part       :False
2021-03-23 14:33:58,584 INFO     strict_rel_part     :False
2021-03-23 14:33:58,585 INFO     num_workers         :8
2021-03-23 14:33:58,585 INFO     num_test_proc       :2
Process Process-1:
Traceback (most recent call last):
  File "/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "ogb/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/models/pytorch/tensor_models.py", line 133, in decorated_function
    raise exception.__class__(trace)
RuntimeError: Traceback (most recent call last):
  File "ogb/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/models/pytorch/tensor_models.py", line 121, in _queue_result
    res = func(*args, **kwargs)
  File "ogb/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/train_pytorch.py", line 228, in train_mp
    train(args, model, train_sampler, valid_samplers, test_samplers, rank, rel_parts, cross_rels, barrier)
  File "ogb/examples/lsc/wikikg90m/dgl-ke-ogb-lsc/python/dglke/train_pytorch.py", line 85, in train
    model.transform_net = model.transform_net.to(th.device('cuda:' + str(gpu_id)))
  File "/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 673, in to
    return self._apply(convert)
  File "/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 387, in _apply
    module._apply(fn)
  File "/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 409, in _apply
    param_applied = fn(param)
  File "/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 671, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
  File "/home/yhc/.conda/envs/ogb-lsc/lib/python3.7/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. To use CUDA with "
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

[hyren]

Hi, the code was tested on pytorch 1.7.0 with cuda 10.1, can you see whether you still encounter this issue with the same setting? Thanks.

[yihong-chen]

Thanks @hyren. It works under pytorch 1.7.0 with cuda 10.1 🚀