Design/Handling of dimension tags

[albertz]

Like batch dim, spatial dims (with dynamic lengths), or static dims (named, or also unnamed).

Dim (earlier DimensionTag) in RETURNN. Directly use that, or wrap it somehow?

Should this (the batch dim) include beam information, or be separate from this?

Relevant for all layers which define some shape or dimension (e.g. Const, Variable).

Should this be enforced, i.e. no simple int allowed in n_out or so but always a Dim object?
And maybe better use out_dim instead of n_out (consistent with rwth-i6/returnn#597).
Edit: It was decided to make nn.Dim mandatory, and use out_dim instead of n_out.

Very related is this issue on RETURNN side on explicit dim tags: rwth-i6/returnn#597

Related is also whether we want unique dim tags? (#48, rwth-i6/returnn#632)

---

This issues covers multiple aspects:

• Use dim tags. Directly use RETURNN Dim. We inherit all its logic on equality etc. We also have FeatureDim, SpatialDim from RETURNN.
• Dim tags (Dim instances) are mandatory for any shape or size
• Shape for all tensors is always available (via Make shape and dims available? #47)
• We have Tensor.verify_out_shape for easy verification
• Solution for in_dim == out_dim, square matrices, Design/Handling of dimension tags #17 (comment), Solution for ambiguous dim tags returnn#871
• Instead of out_spatial_dim argument, the a new spatial dim gets returned. See pool1d for example.
• Tensor shape annotations, moved to Tensor type annotations #97

[Atticus1806]

Would this replace the possibility to define out_type (which you would do in RETURNN)? I didn't continue with the debugging yet, but I think I got some kind of related issue in my config right now. Cannot tell for sure, will report back once I had time to check.

[albertz]

I'm not sure yet whether this would at the same time remove the old way, or just be an additional way.

In any case, I don't think this would really change much in behavior, depending on what you did. So whatever problem you had, you might have it again now. But not sure about the details on your issue. You maybe should report a separate issue about it. But also not sure if this is even related here to returnn-common, or just RETURNN itself?

But also, in general, you never should use out_type anyway. If you do, this sounds like you probably should do it in a different way.

The dimension tags would be used whenever you must specify an axis (for Reduce, Dot, etc), or a new shape (for Const, Variable, etc).

In general, I want to push the concept of named tensors (named array axes; i.e. dimension tags in RETURNN) much more. I want to deprecate or even not allow axis specified as integers. Axis specified as integers (index) can very easily become a problem because the index can easily change (because we optimize sth differently, then the order of axis will be different, or the user adds some other layers and doesn't notice, etc). So the order of axes should not matter at no point.

[Atticus1806]

But also, in general, you never should use out_type anyway. If you do, this sounds like you probably should do it in a different way.

Without further investigation on my own config, how would this work with an Eval layer here? Is returnn able to (or should be able to) implicitly solve the eval and handle the dimension?

In general, I want to push the concept of named tensors (named array axes; i.e. dimension tags in RETURNN) much more.

I like this idea, this would make things a lot more straight forward too, I think.

[albertz]

But also, in general, you never should use out_type anyway. If you do, this sounds like you probably should do it in a different way.

Without further investigation on my own config, how would this work with an Eval layer here? Is returnn able to (or should be able to) implicitly solve the eval and handle the dimension?

The EvalLayer should never change the shape. EvalLayer was never intended for this (although I know that some people abuse it for that). All reshaping logic should be handled by other layers. If you think sth is missing in RETURNN, open a separate issue in RETURNN about this.

But for returnn-common, we don't even need Eval at all because any possible use case is much more elegant using directly the returnn-common features. The possible use cases was actually just convenience, to write sth like a * b + c * d in a simpler way. Although that was already possible before in RETURNN using CombineLayer with kind mul and add. But just a bit annoying to write a lot of separate layers for that. But now in returnn-common, you can simply directly write sth like a * b + c * d. So no need for Eval.

If sth is not clear about this, please open a separate issue, as this is all off-topic now from this issue here.