Skip to content

Gyan output is empty apart from the headers #4

@rahul003

Description

@rahul003

I'm starting my job as follows

LD_PRELOAD=~/mpi-tools/mpi_t/gyan/libgyan.so mpirun -np 8 -hostfile hosts -mca plm_rsh_no_tree_spawn 1 -bind-to socket -map-by slot -x HOROVOD_HIERARCHICAL_ALLREDUCE=1 -x HOROVOD_FUSION_THRESHOLD=16777216 -x NCCL_MIN_NRINGS=4 -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib -x TF_CPP_MIN_LOG_LEVEL=0 python -W ignore train_imagenet_resnet_hvd.py --synthetic -b 256 --num_batches 200 --clear_log

Output is

----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions