Fix recompile from scratch #6036

jackalcooper · 2021-08-25T06:24:45Z

No description provided.

daquexian · 2021-08-25T07:06:34Z

cmake/cfg.cmake

      ${CFG_ARGS}
      --of_cfg_proto_python_dir=${of_cfg_proto_python_dir}
      --project_build_dir=${PROJECT_BINARY_DIR}
      --generate_file_type=cfg.cpp
-    COMMAND ${Python_EXECUTABLE} ${TEMPLATE_CONVERT_PYTHON_SCRIPT}
+    COMMAND ${CFG_Python_EXECUTABLE} ${TEMPLATE_CONVERT_PYTHON_SCRIPT}


cmake/functional.cmake 里也用到了 Python_EXECUTABLE 来生成文件，是不是需要一起改下，这样的话 CFG_Python_EXECUTABLE 就要改个名字了

daquexian · 2021-08-25T07:07:03Z

cmake/cfg.cmake

      ${PY_CFG_ARGS}
      --of_cfg_proto_python_dir=${of_cfg_proto_python_dir}
      --project_build_dir=${PROJECT_BINARY_DIR}
      --generate_file_type=cfg.pybind.cpp
-    DEPENDS ${Python_EXECUTABLE} ${TEMPLATE_FILES} ${CFG_SOURCE_FILE_CONVERT_PROTO} ${PYBIND11_FILE_CONVERT_PROTO}
+    DEPENDS ${TEMPLATE_FILES} ${CFG_SOURCE_FILE_CONVERT_PROTO} ${PYBIND11_FILE_CONVERT_PROTO}


${CFG_Python_EXECUTABLE} 就可以仍然作为依赖了 😂

jackalcooper · 2021-08-25T08:30:34Z

note: 切换 python 版本后，只会编译这些东西。解决了 cfg 和 functional 文件重新编译

[  0%] Built target protobuf
[  0%] Built target make_pyproto_dir
[  0%] Built target of_git_version_create_dir
[  0%] Generating of_git_version/_version.cpp
[  0%] Built target prepare_oneflow_third_party
[  0%] Built target of_git_version
[  0%] Built target oneflow_deps
[ 11%] Built target of_protoobj
[ 14%] Built target of_cfgobj
[ 14%] Built target of_functional_obj
-- Generating pip version: 0.5.0+cpu.git.fcaf0e226, writing to: /home/caishenghang/oneflow/python/oneflow/version.py
[ 14%] Built target of_pyscript_copy
[ 88%] Built target of_ccobj
Scanning dependencies of target of_pyext_obj
[ 88%] Building CXX object CMakeFiles/of_pyext_obj.dir/oneflow/extension/python/numpy.cpp.o
[ 88%] Building CXX object CMakeFiles/of_pyext_obj.dir/oneflow/extension/python/py_compute.cpp.o
[ 88%] Building CXX object CMakeFiles/of_pyext_obj.dir/oneflow/extension/python/py_kernel_caller.cpp.o
[ 88%] Building CXX object CMakeFiles/of_pyext_obj.dir/oneflow/extension/python/py_kernel_registry.cpp.o
[ 93%] Built target oneflow_testexe
[ 93%] Linking CXX shared library libof_pyext_obj.so
[ 93%] Built target of_pyext_obj
Scanning dependencies of target oneflow_internal
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/job_conf.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/register/logical_blob_id.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/placement.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/scope.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/mirrored_parallel.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/framework/user_op_attr.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/operator/op_attribute.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/framework/user_op_conf.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/operator/op_node_signature.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/sbp_parallel.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/parallel_conf_signature.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/initializer_conf.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/regularizer_conf.cfg.pybind.cpp.o
[ 93%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/job/learning_rate_schedule_conf.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/common/data_type.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/common/device_type.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/operator/interface_blob_conf.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/common/shape.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/register/blob_desc.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/register/pod.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/core/operator/op_conf.cfg.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/functional/functional_api.yaml.pybind.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/autograd/autograd.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/autograd/autograd_mode.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/autograd/function_node.cpp.o
[ 94%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/blocking.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/calibration/calibration.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/common.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/deprecated.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/eager/multi_client.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/eager/single_client.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/env/env.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/doc.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/exception/exception.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/flags.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/attr_map.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/blob_register.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/device.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/eager_blob_util.cpp.o
[ 95%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/dtype.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/foreign_boxing_util.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/foreign_callback.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/foreign_watcher.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/id_util.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/instruction_replay.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/instructions_builder.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/job_instance.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/framework.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/nn_graph.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/object.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/object_storage.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/op_arg_util.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/py_remote_blob.cpp.o
[ 96%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/op_builder.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/op_expr.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/random_generator.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/scope_util.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/parallel_conf_util.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/session_util.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/py_kernel_registry.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/shape.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/shut_down_util.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/tensor.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/framework/tensor_tuple.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/functional/common.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/functional/indexing.cpp.o
[ 97%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/functional/py_function.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/functional/python_arg.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/functional/python_functions.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/gil_foreign_lock_helper.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/init.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/job_build/job_build_and_infer.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/job_build/lazy_mode.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/of_api_registry.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/ofblob/ofblob.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/profiler.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/op/op_mgr.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/registry/registry.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/rpc/ccl.cpp.o
[ 98%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/rpc/consistent_rpc_token_scope.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/rpc/rank_group.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/session/session.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/job_conf_symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/op_conf_symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/op_node_signature_symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/placement_symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/sbp_symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/scope_symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/string_symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/symbol/symbol.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/oneflow/api/python/vm/id_generator.cpp.o
[ 99%] Building CXX object CMakeFiles/oneflow_internal.dir/tools/cfg/pybind_module_registry.cpp.o
[ 99%] Linking CXX shared module /home/caishenghang/oneflow/python/oneflow/_oneflow_internal.cpython-37m-x86_64-linux-gnu.so
[100%] Built target oneflow_internal
[100%] Built target of_include_copy
[100%] Built target oneflow_py

…ow-Inc/oneflow into fix_recompile_from_scratch

This reverts commit d6440b2.

github-actions · 2021-08-25T16:48:47Z

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 141.2ms (= 7062.2ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 128.3ms (= 6413.0ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.10 (= 141.2ms / 128.3ms)

PyTorch resnet50 time: 87.0ms (= 4351.0ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 74.5ms (= 3724.0ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.17 (= 87.0ms / 74.5ms)

PyTorch resnet50 time: 57.6ms (= 2878.9ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 47.3ms (= 2366.4ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.22 (= 57.6ms / 47.3ms)

PyTorch resnet50 time: 48.0ms (= 2402.2ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 38.9ms (= 1946.1ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.23 (= 48.0ms / 38.9ms)

PyTorch resnet50 time: 41.8ms (= 2089.7ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 37.8ms (= 1887.5ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.11 (= 41.8ms / 37.8ms)

jackalcooper added 7 commits August 25, 2021 11:04

fix

4b33307

fix

aa50a1c

refine

66d4201

refine

07bed73

refein

df40149

refine

22dc800

refein

6cf7c09

jackalcooper added automerge bug build labels Aug 25, 2021

refei

e406d78

daquexian reviewed Aug 25, 2021

View reviewed changes

jackalcooper added 2 commits August 25, 2021 15:55

refein

7cf32a8

refein

fcaf0e2

jackalcooper added need-simple-ci need-single-client-tests labels Aug 25, 2021

daquexian approved these changes Aug 25, 2021

View reviewed changes

Merge branch 'master' into fix_recompile_from_scratch

215cc8a

oneflow-ci-bot self-requested a review August 25, 2021 08:27

jackalcooper added 3 commits August 25, 2021 16:46

refein

d6440b2

Merge branch 'fix_recompile_from_scratch' of https://github.com/Onefl…

be2a93a

…ow-Inc/oneflow into fix_recompile_from_scratch

Revert "refein"

f633c4a

This reverts commit d6440b2.

oneflow-ci-bot removed their request for review August 25, 2021 09:40

oneflow-ci-bot added 2 commits August 25, 2021 17:40

Merge branch 'master' into fix_recompile_from_scratch

7ad9b47

Merge branch 'master' into fix_recompile_from_scratch

bce0be7

oneflow-ci-bot self-requested a review August 25, 2021 11:27

Merge branch 'master' into fix_recompile_from_scratch

bc01726

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 25, 2021 12:33

Merge branch 'master' into fix_recompile_from_scratch

647c713

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 25, 2021 14:38

oneflow-ci-bot removed their request for review August 25, 2021 16:59

oneflow-ci-bot merged commit a1a9fd9 into master Aug 25, 2021

oneflow-ci-bot deleted the fix_recompile_from_scratch branch August 25, 2021 17:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix recompile from scratch #6036

Fix recompile from scratch #6036

jackalcooper commented Aug 25, 2021

daquexian Aug 25, 2021

daquexian Aug 25, 2021 •

edited

Loading

jackalcooper commented Aug 25, 2021

github-actions bot commented Aug 25, 2021

Fix recompile from scratch #6036

Fix recompile from scratch #6036

Conversation

jackalcooper commented Aug 25, 2021

daquexian Aug 25, 2021

Choose a reason for hiding this comment

daquexian Aug 25, 2021 • edited Loading

Choose a reason for hiding this comment

jackalcooper commented Aug 25, 2021

github-actions bot commented Aug 25, 2021

daquexian Aug 25, 2021 •

edited

Loading