[MXNET-16] Move mshadow/ps-lite/dlpack to 3rdparty #10132

[eric-haibin-lin]

Description

(Brief description on what this PR is about)

Checklist

Essentials

• Passed code style checking (make lint)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[eric-haibin-lin]

@cjolivier01 @szha @marcoabreu please review this PR. Moving nnvm & dmlc-core is much more tricky as many places in amalgamation will be broken. That will be addressed in another PR.

[eric-haibin-lin]

@mbaijal please review license changes

[mbaijal]

@eric-haibin-lin
dlpack and mshadow should still be included in the LICENSE file. Only the paths to the license txt should change.

[eric-haibin-lin]

@mbaijal I see. will do

[marcoabreu]

Could you elaborate on the Amalgamation part? If it breaks due to this change, why does CI pass?

[eric-haibin-lin]

@marcoabreu This PR doesn't move nnvm nor dmlc-core. I attempted to move them in #10122 but amalgamation would break.

[marcoabreu]

Ah I see

[eric-haibin-lin]

@marcoabreu does CI test CMake with dist-kvstore=ON? @rahul003 complained that this PR breaks cmake build

[rahul003]

@eric-haibin-lin Sorry looks like I was wrong. It is fine. I'm not sure what happened when I tried earlier.

But I just looked at CI commands and wow none of the environments build with USE_DIST_KVSTORE. This leaves part of the codebase untested.
@marcoabreu IMO we should enable this for all of them. What do you think @eric-haibin-lin ?
This build flag requires a few dependencies. So that needs to be tested.

Am I looking at the right commands (ci/docker/runtime_functions.sh)?

[marcoabreu]

@rahul003 yes, these are the right commands. Feel free to add it. Just for clarification: This does not require multiple GPUs or nodes, right? What hardware requirements does it have?

Does the behaviour change if that flag is enabled or is just an additional feature? In case of the former, please only add it to a few builds. In case of the latter, feel free to add it to all test cases.