src/executor/graph_executor.cc:1847: Check failed: arg_names.size() == in_args_map.size() (2 vs. 1)

[AlexChalk]

Description

Running lein test in the clojure package directory results in two occurrences of the following error, causing the test suite to fail: org.apache.mxnet.MXNetError: [01:57:41] src/executor/graph_executor.cc:1847: Check failed: arg_names.size() == in_args_map.size() (2 vs. 1)

Environment info (Required)

----------Python Info----------
Version      : 3.6.8
Compiler     : GCC 8.0.1 20180414 (experimental) [trunk revision 259383
Build        : ('default', 'Jan 14 2019 11:02:34')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
No corresponding pip install for current python.
----------MXNet Info-----------
No MXNet installed.
----------System Info----------
Platform     : Linux-4.15.0-58-generic-x86_64-with-Ubuntu-18.04-bionic
system       : Linux
node         : ubuntu-c-4-8gib-tor1-01
release      : 4.15.0-58-generic
version      : #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           4
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               79
Model name:          Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
Stepping:            1
CPU MHz:             2599.996
BogoMIPS:            5199.99
Virtualization:      VT-x
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            40960K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0030 sec, LOAD: 0.3819 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0009 sec, LOAD: 0.1919 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0009 sec, LOAD: 0.2619 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0006 sec, LOAD: 0.3695 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0008 sec, LOAD: 0.0332 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0003 sec, LOAD: 0.0335 sec.
----------Environment----------

Package used (Python/R/Scala/Julia): Clojure

For Scala user, please provide:

1. Java version: (java -version)

openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

2. Maven version: (mvn -version)

Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_222, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "4.15.0-58-generic", arch: "amd64", family: "unix"

3. Scala runtime if applicable: (scala -version)

Scala code runner version 2.11.12 -- Copyright 2002-2017, LAMP/EPFL

Build info (Required if built from source)

Compiler (gcc/clang/mingw/visual studio): n/a (using nightly snapshot)

MXNet commit hash: 7f57e8e10c504fe7a463ba695321b11c6dd4912d

Build config: n/a (using nightly snapshot)

Error Message:

 actual: org.apache.mxnet.MXNetError: [01:57:41] src/executor/graph_executor.cc:1847: Check failed: arg_names.size() == in_args_map.size() (2 vs. 1) :
Stack trace:
  [bt] (0) /tmp/mxnet4687853178926540653/libmxnet.so(+0x2ac5eb) [0x7fb8f7f495eb]
  [bt] (1) /tmp/mxnet4687853178926540653/libmxnet.so(+0x2be36e8) [0x7fb8fa8806e8]
  [bt] (2) /tmp/mxnet4687853178926540653/libmxnet.so(mxnet::Executor::Bind(nnvm::Symbol, mxnet::Context const&, std::map<std::string, mxnet::Context, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::Context> > > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, mxnet::Executor*)+0x7ad) [0x7fb8fa894c3d]
  [bt] (3) /tmp/mxnet4687853178926540653/libmxnet.so(MXExecutorBindEX+0x97b) [0x7fb8fa7cf52b]
  [bt] (4) /tmp/mxnet4687853178926540653/mxnet-scala(Java_org_apache_mxnet_LibInfo_mxExecutorBindEX+0x220) [0x7fb9297a0f00]
  [bt] (5) [0x7fb939018427]

Minimum reproducible example/Steps to reproduce

On a fresh ubuntu server:

wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_core.sh
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_scala.sh
chmod +x ubuntu_core.sh
chmod +x ubuntu_scala.sh
sudo ./ubuntu_core.sh
sudo ./ubuntu_scala.sh
git clone --recursive https://github.com/apache/incubator-mxnet.git apache-mxnet
cd apache-mxnet/contrib/clojure-package/
sed -i 's/\[org.apache.mxnet\/mxnet-full_2.11 "INTERNAL"\]/;\[org.apache.mxnet\/mxnet-full_2.11 "INTERNAL"\]/g' project.clj
sed -i 's/;\[org.apache.mxnet\/mxnet-full_2.11-linux-x86_64-cpu "<insert-snapshot-version>"\]/\[org.apache.mxnet\/mxnet-full_2.11-linux-x86_64-cpu "1.6.0-SNAPSHOT"\]/g' project.clj
sudo apt install leiningen
lein test

What have you tried to solve it?

Reverting to latest official release (version 1.5.0) solves the issue:

wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_core.sh
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_scala.sh
chmod +x ubuntu_core.sh
chmod +x ubuntu_scala.sh
sudo ./ubuntu_core.sh
sudo ./ubuntu_scala.sh
git clone --recursive https://github.com/apache/incubator-mxnet.git apache-mxnet
cd apache-mxnet
git checkout 1.5.0 -b my_mxnet
git submodule update --init --recursive
cd contrib/clojure-package
sed -i 's/\[org.apache.mxnet\/mxnet-full_2.11 "INTERNAL"\]/;\[org.apache.mxnet\/mxnet-full_2.11 "INTERNAL"\]/g' project.clj
sed -i 's/;\[org.apache.mxnet\/mxnet-full_2.11-linux-x86_64-cpu "<insert-snapshot-version>"\]/\[org.apache.mxnet\/mxnet-full_2.11-linux-x86_64-cpu "1.5.0"\]/g' project.clj
sudo apt install leiningen
lein test

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended label(s): Bug

[ZhennanQin]

Hi @adc17 , probably caused by enabling MKLDNN subgraph backend by default on master. Subgraph assumes that the input variables have unique name, if not, the error message you provided will be shown. I don't have the environment to run clojure-package, but I guess it's caused by the improper test cases that use same name for multiple variables. May I know the case names that failed? Then I can have a look for the test to verify my suspect.

[AlexChalk]

@ZhennanQin The Clojure test cases that failed are test-maximum and test-minimum: https://github.com/apache/incubator-mxnet/blob/6de684825130c28dfa75f2a707aeaed64a4340e5/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj#L409-L441

Changing the duplicate "data" strings to "data1" and "data2" resolves the failures 🎉.

@gigasquid I think the correct fix is to update the tests + add an error message for this scenario? I will submit a PR if you agree.

[gigasquid]

@adc17 looks reasonable to me. Thanks for helping on this 💯

[AlexChalk]

I had a look this evening; if we wanted to make the error message clearer it would be DRYest to throw something in the scala stack.

I think it's just createFromListedSymbols and createFromNamedSymbols that would need patching:

https://github.com/apache/incubator-mxnet/blob/3ff1d8794c432142999199df77963ace161ff493/scala-package/core/src/main/scala/org/apache/mxnet/Symbol.scala#L1210-L1213

https://github.com/apache/incubator-mxnet/blob/3ff1d8794c432142999199df77963ace161ff493/scala-package/core/src/main/scala/org/apache/mxnet/Symbol.scala#L1250-L1252

Is this justified in the name of a clearer error message (I'm thinking something like "comparing two values with the same label is not supported")? The tradeoffs are maintainability + I think it would have to hit people without the MKLDNN subgraph backend enabled too.

In the meantime I will submit a simple PR to just change the labels used in the tests ;).

[gigasquid]

@lanking520 Thoughts on the Scala side above? ^

[lanking520]

Sure, please do a PR in there. I also find some tricks we can do to automatically figure out the data names:

• LIB.NNSymbolListInputNames will bring you all names in the Symbol including the input
• You can find all names for Parameter while loading.

The subtraction for the above two will be the input names.

[samskalicky]

@zachgk assign [@samskalicky ]

[AlexChalk]

@lanking520 I'm wondering if console output would be better than actually throwing an error here, that way users without the MKLDNN subgraph backend enabled don't encounter an unnecessary exception?