Represent operators' inputs, outputs, and attributes by Variable

[wangkuiyi]

I am thinking of operator attributes as variables. Considering that class Variable can hold any typed value, and there isn't any reason that forbids operations from setting the value of other operations' attributes, I tend to think attributes, like inputs and outputs, should be represented by variables.

[jacquesqiao]

1. Variables are within scope, and there may be multiple scopes, so if we use the attribute as Variable, we should init it in every running scope.
2. After discussing with yuyang, I know that attributes of an operator are const and will not change, so if it reasonable to use attr as Variable?

[reyoung]

It seems that there is some difference between Variable and Attribute.

1. The attribute is a const private data member in each Op instances.

   • Attributes must pass to Op when creation.
   • It will not be shared between Op instances. To store it in a Scope is not necessary.
   • It is const, not mutable.
   • It stores only in CPU not GPU.

2. The attribute should have many constraints or validators, e.g., Required, DefaultValue, LargerThan.

   • Validation between inputs/outputs and attributes are different. Validation of inputs/outputs should be ShapeMismatch, MustOnGpu, etc.
   • Run must validate all inputs and outputs each time, but it is no need parsing Attribute every time.

3. In concept, Attribute of Op is part of topology, not Scope. Suppose we have a FcOp, and FcOp has an attribute named activation. It is obvious that activation should be part of the topology. Another example is step_net for RNN Op.

So I think an attribute should not be a normal variable. But maybe we could pass an independent Scope while creating an Op. Use that scope for the argument of Op creation.

OperatorBase* CreateOperator(std::vector<string> inputs, std::vector<string> outputs, std::string type, const Scope& attrs) {

}

[jacquesqiao]

What about some mutable states such as momentum? How to init them and store them.
Maybe these value can be used as Variable.

[reyoung]

What about some mutable states such as momentum? How to init them and store them.

Momentum is NOT a configurable value for Op. Momentum is just like Gradient which is calculated in runtime.

[emailweixu]

There are two types of attributes. The first type cannot be changed for a given model (e.g., the size of fc layer). The second type can be changed (e.g., the scale for cosine similarity). I think whenever possible, we should put the second type of attributes as part of inputs. This can make the framework more flexible, because it has the potential to allowing those attributes to be dynamically changed. For the first type of attributes, they have to be given when the model is defined. So they have to be handled differently as inputs, though we still possible to use variable to store it.