Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Hardtanh activation #4049

Merged
merged 16 commits into from
Jan 6, 2021
Merged

Add Hardtanh activation #4049

merged 16 commits into from
Jan 6, 2021

Conversation

MARD1NO
Copy link
Contributor

@MARD1NO MARD1NO commented Dec 28, 2020 •
edited
Loading

概述

增加 hardtanh 算子
支持inplace操作
图片

功能 CheckList

模板中自带的复选框可留空，但是不能删除。可根据实际情况增加复选框选项。

Op

  • Op SetBatchAxisInferFn
  • Op SetGetSbpFn
  • Op SetInputArgModifyFn
  • Op 反向梯度注册

Kernel

仅注册浮点数float16/32/64类型

  • CPU in:float32

  • CPU in:float64

  • GPU in:float32

  • GPU in:float64

  • GPU in:float16

Python Wrapper

  • Python API 参数检查及异常提示
  • 接口注释
  • Example 

测试

  • 单机单卡 CPU Test Case
  • 单机单卡 GPU Test Case
  • 单机多卡 CPU Test Case
  • 单机多卡 GPU Test Case
  • 分布式 CPU Test Case
  • 分布式 GPU Test Case

GPU 有效带宽

理论带宽, 环境使用的是本地机器 2060Super

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(GB/s)
   32000000			378.4

实际带宽


I1229 16:34:10.009622 57577 kernel.cpp:89] PROFILER::KERNEL::CUDA_MEMORY_BANDWIDTH op_name: Hardtanh_1 elapsed(ms): 0.182656 memory_size(Byte): 50331648 bandwidth(GB/s): 256.63

PR Checklist

  • PR 标题语句通畅，明确表达 PR 内容，适合直接作为新版本发布时的 changelog
  • 代码格式化
  • 已经本地编译通过
  • 已本地针对改动测试
  • 已添加 type 标签:(填写 type 标签名，如 bug, enhancement, purge, feature, documentation)
  • 已添加 component 标签:(填写 component 标签名，如 op, system, eager, build, xla, python, ci, test, tooling, onnx)
  • Draft 转正式 PR 前已请人 Review

@jackalcooper jackalcooper changed the title Dev Hardtanh Add Hardtanh Op Dec 29, 2020
@MARD1NO MARD1NO requested a review from doombeaker December 29, 2020 05:27
@Ldpe2G Ldpe2G marked this pull request as ready for review December 31, 2020 08:54
@oneflow-ci-bot oneflow-ci-bot removed their request for review December 31, 2020 09:44
@oneflow-ci-bot oneflow-ci-bot removed their request for review December 31, 2020 15:07
@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot January 6, 2021 08:16
@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot January 6, 2021 08:45
@oneflow-ci-bot oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot January 6, 2021 08:48
@MARD1NO MARD1NO requested a review from oneflow-ci-bot January 6, 2021 12:01
@oneflow-ci-bot oneflow-ci-bot merged commit 512b7c6 into master Jan 6, 2021
@oneflow-ci-bot oneflow-ci-bot deleted the dev_hardtanh branch January 6, 2021 14:34
liujuncheng pushed a commit that referenced this pull request Jun 3, 2021
@MARD1NO @doombeaker @oneflow-ci-bot
Add Hardtanh activation (#4049)
06095ed 
* add cpu version

* add GPU version

* refine namespace

* add test case

* add docs

* Update oneflow/user/kernels/hardtanh_kernel.cpp

Co-authored-by: doombeaker <later@usopp.net>

* fix

* fix to formula in infer grad formula

* delete int type

* fix format

Co-authored-by: doombeaker <later@usopp.net>
Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com>
Former-commit-id: 512b7c6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@doombeaker doombeaker doombeaker left review comments

@Ldpe2G Ldpe2G Ldpe2G approved these changes

@oneflow-ci-bot oneflow-ci-bot Awaiting requested review from oneflow-ci-bot

Assignees
No one assigned
Labels
automerge enhancement op
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@MARD1NO @Ldpe2G @doombeaker @oneflow-ci-bot