Skip to content

Commit

Permalink
[one::OpBuilder] Contruct variable and user op from Python (#4458)
Browse files Browse the repository at this point in the history 
* use symbol::Storage<OperatorConfSymbol>

* _NewOpKernelObject

* mig OpKernelObject

* mig object_storage

* make of_format

* del comment

* std::function<void(Object*)

* mig NewOpKernelObject and _StatefulCallOpKernel

* mig _StatefulCallOpKernel and GetSharedOpKernelObject4ParallelConfSymbol

* del object_storage.cpp

* use name GLOBAL_PARA_SYM2SHARED_OPKENEL_OBJ_MUTEX

* mig CheckRefInBlobObjectParallelDesc and  OperandBlobObjects rel api

* mig _StatelessCall

* mig _StatelessCall

* [one::OpBuilder] Refactor Operation.

* mig StatelessCall api

* mig StatefulCall

* mig callback api

* mig MakeLazyRefBlobObject

* refactor CudaHostPinBlob

* sort out InstructionsBuilder api

* [one::OpBuilder] Refine

* mig PhysicalRun and LogicalRun

* use oneflow_api.deprecated.LogicalRun & PhysicalRun

* delete vm_util.py

* change FindOrCreateDelegateBlobObject args

* add SetShuttingDown

* rm python_interpreter_util.py

* add blank line

* mig BlobCache

* use FindOrCreateDelegateBlobObject in c++

* refactor session_util.cpp

* use IsShutDown

* refactor BlobRegister

* [one::OpBuilder] Refactor OpExpr and OpExprInterpreter.

* [one::OpBuilder] Remove member function evaluate.

* [one::OpBuilder] Remove OpInterpreter to facilitate CR.

* [one::OpBuilder] Refine

* fix distribute test exit bug

* del comment

* [one::OpBuilder] Add more op exprs.

* mig id_util and scope_util

* use cfg_op_conf and Object*

* use Object*

* del _

* fix func name error

* [one::OpBuilder] Refine

* [one::OpBuilder] Modify op input names by InOutBnAccessor.

* [one::OpBuilder] Fix op expr python api.

* use MapAt and shared_ptr

* [one::OpBuilder] Using indexed ibns and obns instead of tensor names.

* [one::OpBuilder] Update

* [one::OpBuilder] Complete op interpreter in the main.

* use shared_ptr or const ref

* minor fix

* add todo

* [one::OpBuilder] Export and extend BoxingUtil in python.

* [one::OpBuilder] Support variable op interpretation.

* [one::OpBuilder] Refine

* [one::OpBuilder] Refine placeholder prefix.

* minor fix

* minor djustment

* minor fix

* [one::OpBuilder] Migrate snapshot manager.

* minor optimize

* minor fix

* minor optimize

* [one::OpBuilder] Migrate Session.

* [one::OpBuilder] Fix return type.

* [one::OpBuilder] Refine variable interpretation.

* minor optimize

* fix bug

* [one::OpBuilder] Call python FeePath.

* [one::OpBuilder] Fix typo.

* minor fix

* [one::OpBuilder] Fix merge bugs.

* [one::OpBuilder] Bugfix.

* [one::OpBuilder] Refine

* [one::OpBuilder] Bugfix

* [one::OpBuilder] Refine

* [one::OpBuilder] Fix typo.

* [one::OpBuilder] Fix placeholder prefix of op builer.

* [one::OpBuilder] Set output blob object after running the instruction.

* [one::OpBuilder] Remove TensorNameScope temporarily.

* [one::OpBuilder] Fix api.

* [one::OpBuilder] Add TensorNameScope.

* [one::OpBuilder] Modify the op builder apis to return Maybe type.

* [one::OpBuilder] Remove unused header.

* [one::OpBuilder] Fix

* [one::OpBuilder] Fix typo

* [one::OpBuilder] Fix bugs.

* [one::OpBuilder] Create tensor from blob object.

* [one::OpBuilder] Return Maybe in op interpreter.

* [one::OpBuilder] Fix merge conflicts and reformat.

* [one::OpBuilder] Remove unused code.

* [one::OpBuilder] Go through eager mode.

* [one::OpBuilder] Fix typo

* [one::OpBuilder] Refine

* [one::OpBuilder] Remove redundant file system and refine code style.

* [one::OpBuilder] Use TensorTuple instead of TensorList and provide method to access the valide interpreter.

* [one::OpBuilder] Fix bug

* [one::OpBuilder] Remove state input.

* [one::OpBuilder] Create output tensors for lazy mode.

* [one::OpBuilder] Change pybind hold type to shared_ptr for Callback, Watch and BoxingUtil.

* [one::OpBuilder] Move to deprecated.

* [one::OpBuilder] Remove unused code.

* [one::OpBuilder] Fix typo.

* [one::OpBuilder] Support TensorTuple input and fix typo.

* [one::OpBuilder] Fix bugs.

* [one::OpBuilder] Change pybind hold type to shared_ptr for Callback, Watch and BoxingUtil.

* [one::OpBuilder] Fix bugs.

* [one::OpBuilder] Migrate snapshot manager.

* [one::OpBuilder] Remove redundant file system and refine code style.

* [one::OpBuilder] Migrate Session.

* [one::OpBuilder] Refine

* [one::OpBuilder] Refine

* [one::OpBuilder] Call python FeePath.

* [one::OpBuilder] Export and extend BoxingUtil in python.

* [one::OpBuilder] Deprecate boxing util python api.

* [one::OpBuilder] Reformat

* Merge

* Refine

* Use reference op expr.

* Fix default value

* Fix LocalFS use bug

* Fix and refine

* Return Maybe

* Reformat

* Refine

* Remove initialization to simplify variable op interpretation.

* Revert SnapshotManager and Session

* Fix register BoxingUtil

* Remove unused header

* Revert foreign callback

* Refine code style

* Fix register BoxingUtil

* Get device from parallel desc to make a new tensor.

* Boxing call instructions and refine.

* Boxing call instructions and refine.

* Construct variable and user op from Python.

* Construct variable and user op from Python.

* Fix ci error

* Swap the op conf since the move constructor was not provided by protobuf messages.

* Refine

* Refine code style

* Reformat

* Fix error merge

* Make interpreter stateless and remove OpExprInterpContext.

* Refine

* reformat

* Refine

Co-authored-by: clackhan <han_binbin@163.com>
Co-authored-by: hjchen2 <hjchen2>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
  • Loading branch information
@hjchen2 @clackhan @oneflow-ci-bot
3 people authored Mar 26, 2021
1 parent 4380494 commit 1117632
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 7 deletions.
1 change: 1 addition & 0 deletions cmake/cfg.cmake
View file
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ function(GENERATE_CFG_AND_PYBIND11_CPP SRCS HDRS PYBIND_SRCS ROOT_DIR)
oneflow/core/job/job_conf.proto
oneflow/core/job/placement.proto
oneflow/core/framework/user_op_attr.proto
oneflow/core/framework/user_op_conf.proto
oneflow/core/job/sbp_parallel.proto
oneflow/core/job/scope.proto
oneflow/core/job/mirrored_parallel.proto
Expand Down
32 changes: 25 additions & 7 deletions oneflow/api/python/framework/op_expr.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ limitations under the License.
#include "oneflow/core/framework/op_interpreter_util.h"
#include "oneflow/core/framework/tensor.h"
#include "oneflow/core/framework/tensor_tuple.h"
#include "oneflow/core/framework/user_op_conf.cfg.h"

namespace py = pybind11;

Expand All @@ -47,10 +48,29 @@ Maybe<std::vector<std::shared_ptr<one::Tensor>>> Interpret(
return static_cast<std::shared_ptr<std::vector<std::shared_ptr<one::Tensor>>>>(outputs);
}

template<typename OpT, typename ConfT,
typename std::enable_if<std::is_base_of<one::BuiltinOpExpr, OpT>::value>::type* = nullptr>
void PybindExportOpExpr(py::module& m, const char* op_type_name) {
using ProtoConfT = decltype(std::declval<OpT>().proto());
py::class_<OpT, one::BuiltinOpExpr, std::shared_ptr<OpT>>(m, op_type_name)
.def(py::init([](const std::string& op_name, const std::shared_ptr<ConfT>& op_conf,
const std::vector<std::string>& indexed_ibns,
const std::vector<std::string>& indexed_obns) {
typename std::decay<ProtoConfT>::type proto_op_conf;
op_conf->ToProto(&proto_op_conf);
return std::make_shared<OpT>(op_name, std::move(proto_op_conf), indexed_ibns, indexed_obns);
}))
.def_property_readonly("proto",
[](const OpT& op) { return std::make_shared<ConfT>(op.proto()); });
}

} // namespace

ONEFLOW_API_PYBIND11_MODULE("one", m) {
py::class_<one::OpExpr, std::shared_ptr<one::OpExpr>>(m, "OpExpr")
.def_property_readonly("type", &one::OpExpr::type)
.def_property_readonly("input_num", &one::OpExpr::input_num)
.def_property_readonly("output_num", &one::OpExpr::output_num)
.def("apply",
[](const one::OpExpr& op_expr, const std::vector<std::shared_ptr<one::Tensor>>& inputs) {
return Interpret(op_expr, inputs).GetOrThrow();
Expand All @@ -61,14 +81,12 @@ ONEFLOW_API_PYBIND11_MODULE("one", m) {

py::class_<one::BuiltinOpExpr, one::OpExpr, std::shared_ptr<one::BuiltinOpExpr>>(m,
"BuiltinOpExpr")
.def_property_readonly("name", &one::BuiltinOpExpr::op_name);
.def_property_readonly("name", &one::BuiltinOpExpr::op_name)
.def_property_readonly("indexed_ibns", &one::BuiltinOpExpr::indexed_ibns)
.def_property_readonly("indexed_obns", &one::BuiltinOpExpr::indexed_obns);

py::class_<one::UserOpExpr, one::BuiltinOpExpr, std::shared_ptr<one::UserOpExpr>>(m, "UserOpExpr")
.def_property_readonly("type", &one::UserOpExpr::type)
.def_property_readonly(
"proto", [](const one::UserOpExpr& op) { return PbMessage2TxtString(op.proto()); })
.def_property_readonly("indexed_ibns", &one::UserOpExpr::indexed_ibns)
.def_property_readonly("indexed_obns", &one::UserOpExpr::indexed_obns);
PybindExportOpExpr<one::UserOpExpr, cfg::UserOpConf>(m, "UserOpExpr");
PybindExportOpExpr<one::VariableOpExpr, cfg::VariableOpConf>(m, "VariableOpExpr");
}

} // namespace oneflow

0 comments on commit 1117632

Please sign in to comment.