Meta-problem: deciding what class of neural network works best for a given function

[timql]

** not urgent: for down the road

• Train a classifier on a bunch of different functions and how well a range of neural network topologies approximate that function

  • The input to this classifier would be some representation of the function's AST, and the expected output would be which class of neural network should be used to approximate the function. Potentially more powerful than trying to define families of functions/ASTs explicitly.

• We can just generate a whole bunch of small ASTs to use as training data for this classifier

[timql]

Idea:

Since no arbitrary neural network can outperform an arbitrarily deep feedforward network, just use RNNs, and have a scoring function determine how large and/or complex a function is (maybe by looking at the syntax tree). We can then use this "complexity score" to determine how deep and wide the RNN should be. We can also use our library to approximate this scoring function (essentially a regression problem) so as to generalize to arbitrary functions from known simple ones that we generate.

[ch3njust1n]

Potential approach for estimating width and depth of a network for learning a given function:

We could use three neural nets:
Complexity Network:
Input: AST
Output: Complexity score

Depth Network:
Input: AST, complexity score
Output: Estimated total depth - D

Breadth Network:
Input: AST, complexity score, intermediate depth - d (particular layer in range(1, D+1))
Output: Estimated breadth - B, at a given depth

We could generate AST from Python modules and generate LSTM with the NEAT algorithm as training data for the Complexity, Depth, and Breadth networks. More specifically, we can use the NEAT algorithm to evolve a population of LSTM networks for a specific function. We can then select the networks with the best accuracies and use their dimensions as training data for the Depth and Breadth networks. The Depth network could take an AST and the AST's complexity score, and be trained to output the total depth of the network. The estimated total depth, AST, and complexity score could then be used to train the Breadth network to learn the breadth of the network at specific depths. To train the Breadth network, we would iterate from 1 to the estimated depth. On each iteration, we would feedforward the AST, complexity score, and a specific depth, which would be the iteration number. We would then backprop after giving the breadth at that depth, so that it can learn breadth at a given depth for that type of function. After training these networks, we could predict the dimensions for constructing a vanilla LSTM, which can then be used to learn a function.

Let me know what you guys think. I also found a NEAT implementation in Python that I really want to dissect. I ran it and it works for the default use cases from the original paper. If this approach makes sense to everyone else, I'll start digging into it more.

P.S.
The interesting thing, as Tim and I discussed yesterday, is that neural networks are constructed using intuition and it seems that know one has any good reasons nor understanding for why a given depth or a given width allows a network to learn what it learns with whatever accuracy it achieves. Similarly, Demis Hassabis, co-founder of Google DeepMind, has mentioned in interviews that if you ask professional Go players why they made a certain move, they say that "it felt right". Because of this, no one had been able to construct a reasonable value function for previous AI systems for playing Go. What DeepMind did was build a set of neural networks, one for estimating the necessary depth, and one for estimating the necessary breadth of the Monte Carlo Search Tree. Combined these networks captured the intuition of the value function. Perhaps, the same can be done to capture the intuition for a network's dimensions given the complexity of the function one is trying to estimate.

P.S.S.
Another cool idea that could potentially build off of this is building a network that can predict the amount of training data a particular architecture needs to learn a function to a specific accuracy.

P.S.S.S.
We could also use pruning and dropout to generate networks for training instead of NEAT since NEAT would require learning a lot of very deep networks. Pruning and dropout would allow us to reuse an already trained network with a guaranteed accuracy.