Extremely slow network construction (or maybe infinite loop, unclear)

[albertz]

At the same time, memory increases all the time (slowly), so after some minutes, it would crash due to out-of-memory.

This is via a transducer search config generated via returnn-common, this config, but the problem also occurs when I dump it as pure RETURNN net dict, as in this config. Simplified further, this config.

In the log you see this as some of the last messages:

...
layer /lm/lstm/'output': [F'embed'(256),F|F'4*lstm'(4096)] float32
layer /lm/'lstm': [F'embed'(256),F|F'4*lstm'(4096)] float32
layer /lm/'output': [F'embed'(256),F|F'4*lstm'(4096)] float32
layer /'lm': [F'embed'(256),F|F'4*lstm'(4096)] float32
layer /beam_search/'random_4': [F'lstm'(1024),F|F'readout'(1000)] float32
layer /beam_search/'readout_in_lm_weight': [F'lstm'(1024),F|F'readout'(1000)] float32
layer /'readout_in_bias': [F|F'readout'(1000)] float32
layer /beam_search/'random_6': [F'readout//2'(500),F|F'phones'(61)] float32
layer /beam_search/'out_nb_label_logits_weight': [F'readout//2'(500),F|F'phones'(61)] float32
layer /beam_search/'out_nb_label_logits_bias': [F|F'phones'(61)] float32

When looking at the stack trace in a debugger, rec template construction is involved.

[albertz]

To get an impression of the (maybe endless, or very long) loop, in _SubnetworkRecCell._construct_template().RecSubnetGetTemplateLayer.__call__(), add:

        print(" ", self, "-" * len(ConstructCtx.layers), name)

You get some output like:

...
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_bias
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:base:lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/ne
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ---------------- model/combine_bc
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ---------------- base:base:readout_in_bias
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ---------------- model/combine_bc
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ----------------- model/readout_in_am
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ----------------- model/masked_computation/readout_in_lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------ model/masked_computation
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/reinterpret_set_sparse_dim
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_bias
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:base:lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/ne
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/reinterpret_set_sparse_dim
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_bias
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:base:lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/ne
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/reinterpret_set_sparse_dim
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_bias
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:base:lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/ne
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------ model/masked_computation
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------ model/masked_computation
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- model/reinterpret_set_sparse_dim
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------------- base:embed_bias
...

Repeating a lot...

[albertz]

Maybe there is actually some bug in the config, and some exception occurs, however, due to the template construction logic, it keeps trying.

[albertz]

Maybe the flat net construction (#992) would also solve this.

[albertz]

The masked computation layer is not involved. I simplified it further (updated config is in the main post) and remove that part. Now the debug output:

...
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------ base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------ model/out_wb_label_logits/dot
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------ base:out_wb_label_logits_bias
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------ model/out_wb_label_logits/dot
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------- model/reduce_out
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> -------- model/combine_bc
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> --------- model/readout_in_am
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> --------- model/readout_in_lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ---------- model/readout_in_lm/output
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ----------- model/readout_in_lm/dot
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------ model/lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------ base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------ model/lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------ base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------ model/lm
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ------------ base:readout_in_lm_weight
  <_SubnetworkRecCell '/beam_search/loop(rec-subnet)'> ----------- model/readout_in_lm/dot
...

[albertz]

I was thinking about a complete redesign of the net construction (#1128). But the main problem here is actually in the rec layer subnet template construction heuristic (for resolving circular dependencies). This is somewhat independent. Separate issue on redesigning the rec layer subnet template construction heuristic: #1129

[albertz]

My assumption is that some (actual correct) exception occurs in some layer (e.g. model/lm as you see that often last in the debug output) but it keeps trying again and again.

We should try to detect this somehow. But how?

[albertz]

There is a bug in the original code:

    batch_dims = data.batch_dims_ordered(data_spatial_dim)

Should have been:

    batch_dims = data.batch_dims_ordered((data_spatial_dim, data.feature_dim))

But this is maybe then also a problem for returnn-common, that there was no real error at that point.