Dev integrate cfg #3597
Merged
Dev integrate cfg #3597
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…dev_integrate_cfg
…dev_integrate_cfg
…dev_integrate_cfg
…dev_integrate_cfg
…dev_integrate_cfg
jackalcooper
requested changes
Sep 24, 2020
|
请测试两个东西：
|
2 应该没问题，ci里面会测试到 |
1是指的增加或减少下面ALL_CFG_CONVERT_PROTO里的proto文件配置吧 list(APPEND ALL_CFG_CONVERT_PROTO
oneflow/core/common/data_type.proto
oneflow/core/common/device_type.proto
oneflow/core/job/sbp_parallel.proto
oneflow/core/job/mirrored_parallel.proto
oneflow/core/job/scope.proto
oneflow/core/record/record.proto
oneflow/core/record/image.proto
)做了两把实验，增加或减少proto文件配置都不会导致整个 oneflow 重新编译 |
我想的是修改。减少的话，被删除的 proto 原先产生的文件有消失吗。 |
减少的话，被删除的 proto 原先产生的文件不会消失，只是编译的时候不会编译原先产生的源文件而已，这个proto2cpp那边好像也是一样的情况吧。 |
lixinqi
approved these changes
Oct 7, 2020
liujuncheng
pushed a commit
that referenced
this pull request
Jun 3, 2021
* integrate cfg primary * add depend of of_ccobj * add dependence of of_cfgobj * fine-tuning the way of cfg integration * add proto/cfg convert & DebugString * finetune cfg * optimize cfg & add support of nested message * fix a small bug * arrange cfg code * optimize cfg * fix code format * add of_proto_python_dir arg to template_convert.py * fix cfg constructor bug * remove useless line in cfg.cmake * optimize cfg.cmake * delete useless line in cfg.cmake * fintune cfg.cmake * fix a small bug of oneof field * support oneflow_api_registry * fix code format * Expand folder range * optimize of_api_registry * Update of_api_registry.h add a blank line between constructor and member function * update pybind_module_registry * fix code format * optimize oneflow_pybind_api * optimize pybind registry * fix code foemat * optimize cfg * fix oneflow.cmake * add a blank line * optimize cfg Co-authored-by: lixinqi <lixinqi0703106@163.com> Former-commit-id: e01b57a
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cfg设计初衷
在目前oneflow系统中存在大量的通过protobuf定义的结构化数据（message），message的不同语言上的对象在内存上并无联系，oneflow系统中C++和Python之间mesage数据信息交换是通过序列化和反序列化。在eager模式下，oneflow系统会十分频繁的进行前后端数据交互，严重影响eager模式下的执行效率，减小C++和Python之间的信息传递开销显得尤为重要。cfg工具因此而生。
cfg接口
cfg工具将.proto文件编译成.cfg.h和.cfg.cpp文件，.cfg.h提供与.pb.h中C++类中相同的接口（不保证提供全部方法，但在目前的oneflow系统中完全够用），同时会在Python导出同C++端一样的接口。此外，.cfg.h中的C++类可以以.pb.h中的C++类对象为输入构造其对象，并且提供成员方法ToProto(oneflow::Bar* proto_bar)转化为.pb.h中的C++类对象。类似.pb.h中的C++类，.cfg.h中的类提供成员函数 ::std::string DebugString()用于输出类内部信息。
使用方法详见：cfg文档