RNN cell demo with ptb LSTM language model

[pluskid]

As people are asking (#3142) how to use the new RNN symbol in mxnet, we provide this simple example as a demo. More importantly, this allow us to identify the use cases and issues we need to address. Currently we only have a wrapper to cuDNN RNN cell, so it is not available for CPU mode. Also we are working to stabilize the interfaces, as you can see from the comments of the demo, there are still many workarounds need to be applied to be able to work nicely with the rest of the mxnet high level APIs.

@sbodenstein @antinucleon One important issue I need to mention is that currently the LSTM cell takes all the parameters as a concatenated big vector. This is not compatible with our initializer architecture.

I only verified it is able to run and the perplexity is decreasing. But I did not compare with our old LSTM implementation results.

[sbodenstein]

@pluskid: one simple solution (which we use) is to have a high-level function create a view of all the sub-weights and biases (a dictionary of sub-dictionaries, one per layer), and then use this for initialization. If we decide on keeping the single parameter design, I can contribute a Python version of this.

Then for the Python initalizer: it already has special initialization logic for things like BatchNorm, you can have one for the RNN, which calls the extractor function and decides one the init method (like IRNN https://arxiv.org/pdf/1504.00941.pdf etc).

[pluskid]

@sbodenstein I'm OK with a single concatenated parameter, which presumbly should be more efficient. Currently the initializer does not have access to much information, apart from the name and the array itself. How do you plan to extract information used to decide which part is weights and which part is bias?

[sbodenstein]

@pluskid: I've already written a version for Mathematica. Its pretty simple: cuDNN has an API function for splitting this: cudnnGetRNNLinLayerBiasParams + cudnnGetRNNLinLayerMatrixParams. Using this + the docs for this makes it possible to work out whats going on (eg the biases are all at the end of the parameter vector). I then simply write a test making use of the definitions for GRU, LSTM, etc cuDNN uses, and check that my interpretation of the parameters are correct (and ensure that the splitter remains correct for future cuDNN versions).

[pluskid]

@sbodenstein Great! One potential issue with the existing high-level API implementation is that the models tries to allocate the memory for the reported parameters (separately), instead of continuous chunk of concatenated vector. I guess what seems to be a feasible solution is to let the rnn parameters to have a special name (postfix) so that a customized initializer could recognize it, split it accordingly and do proper initialization.