Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

【Hackathon 5th No.33】为 Paddle 新增 atleast_1d / atleast_2d / atleast_3d API -part #58323

Merged
merged 12 commits into from
Nov 16, 2023

Conversation

megemini
Copy link
Contributor

@megemini megemini commented Oct 23, 2023

PR types

New features

PR changes

APIs

Description

【Hackathon 5th No.33】为 Paddle 新增 atleast_1d / atleast_2d / atleast_3d API

RFC: PaddlePaddle/community#679

涉及文件:

  • python/paddle/__init__.py 将 API 暴露出来
  • python/paddle/tensor/__init__.py 将 API 暴露出来
  • python/paddle/tensor/manipulation.py 实现 API
  • test/legacy_test/test_atleast.py 单元测试

本地测试情况:

  • 单元测试通过
  • coverage 100%
  • 示例测试通过

请评审!

@paddle-bot
Copy link

paddle-bot bot commented Oct 23, 2023

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-ci-bot
Copy link

paddle-ci-bot bot commented Oct 31, 2023

Sorry to inform you that 2395977's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`.

Note:
``int8``, ``uint8``, ``complex64``, ``complex128`` are not supported in static graph mode.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为什么在静态图模式下这几个类型不能支持?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这几个方法里面用到了 reshape 将 0-dim 转为 n-dim,当时 rfc 的时候没想到 reshape 的动/静态图支持的不一样,静态图不支持这几个类型:

def reshape(x, shape, name=None):
    if in_dynamic_mode():
        ...
    else:
        check_variable_and_dtype(
            x,
            'x',
            [
                'float16',
                'float32',
                'float64',
                'int16',
                'int32',
                'int64',
                'bool',
                'uint16',
            ],
            'reshape',
        )

所以这里单独将这几个类型单拎出来了 ~

另外,bool 应该也支持,当时 rfc 的时候没有提到 bool 类型,是否也加进去?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reshape 的 check_variable_and_dtype 是否可以加上这几个类型呢?如果可以的话,需要单独提一个PR支持静态图下reshape的 int8, uint8, complex64, complex128类型。

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

看了一下底层 c++ 算子的实现,没找到 reshape 对于这几种数据类型有单独处理。

尝试修改 paddle/tensor/manipulation.py :

def reshape(x, shape, name=None):
    if in_dynamic_mode():
        ...
    elif in_pir_mode():
       ...
    else:
        check_variable_and_dtype(
            x,
            'x',
            [
                'float16',
                'float32',
                'float64',
                'int16',
                'int32',
                'int64',
                'bool',
                'uint16',
                'int8',
                'uint8',
                'complex64',
                'complex128',
            ],
            'reshape',
        )

简单验证:

In [1]: import numpy as np
   ...: import paddle
   ...: 
   ...: paddle.enable_static()
   ...: with paddle.static.program_guard(paddle.static.Program()):
   ...:     feed = {}
   ...:     x = paddle.static.data('x', [12], 'int8')
   ...:     feed['x'] = np.arange(12).astype('int8')
   ...: 
   ...:     out = paddle.reshape(x, [2, 6])
   ...:     exe = paddle.static.Executor(paddle.CUDAPlace(0))
   ...:     res = exe.run(feed=feed, fetch_list=[out])
   ...: 
   ...:     print(res)
   ...: 
[array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11]], dtype=int8)]
  • 未修改之前,会报错 TypeError
  • 修改之后,能够正常运行

另外,in_pir_mode 这里先不修改吧 ~ 我试了一下 os.environ['FLAGS_enable_new_ir_in_executor'] = 'True',接口会挂掉 ... ...

那我这边单独提 PR 再逐个类型验证一下吧 ~

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it !

@megemini
Copy link
Contributor Author

megemini commented Nov 4, 2023

Update 20231104

  • 修改单测文件 test_atleast.pytest_atleast_nd.py
  • 修改 docstring 支持 bool 类型
  • 增加 bool 类型的单测

@megemini
Copy link
Contributor Author

megemini commented Nov 10, 2023

Update 20231110

根据 reshapedtype 修改:

  • docstring 中的支持类型
  • 修改相应的单测

Update 20231111

目前静态图测试都加上了 @test_with_pir_api,此 PR 也加上了,所以需要 reshape 也在 pir 下支持所有数据类型。

#58924 修改 reshapepir 支持的数据类型。

需要等 #58924 确认无误并合入后重跑 CI ~

@luotao1
Copy link
Contributor

luotao1 commented Nov 13, 2023

@megemini 可以merge develop重跑CI了

luotao1
luotao1 previously approved these changes Nov 14, 2023
Comment on lines +115 to +117
from .manipulation import atleast_1d # noqa: F401
from .manipulation import atleast_2d # noqa: F401
from .manipulation import atleast_3d # noqa: F401
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we need to support the usage of x.atleast_1d(), x.atleast_2d(), x.atleast_3d() when x is a tensor ? If so, add them to the list of tensor_method_func below

),
)
class TestAtleastErrorCombineInputs(BaseTest):
"""test combine inputs, like: `at_leastNd((x, y))`, where paddle treats like numpy, not pytorch"""
Copy link
Contributor

@jeff41404 jeff41404 Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please delete not pytorch

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @zhwesky2010 :atleast_xd 处理语义在遇到tuple/list时和numpy对齐,和pytorch有差异,后续在代码转换时可以关注下

Copy link
Contributor Author

@megemini megemini Nov 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @zhwesky2010 :atleast_xd 处理语义在遇到tuple/list时和numpy对齐,和pytorch有差异,后续在代码转换时可以关注下

主要区别是:

  • paddle/numpy -> paddle.atleast_1d((x, y)) 将 (x, y) 作为一个整体输入,并转换为 tensor (此处需要注意 broadcast)
  • pytorch -> torch.atleast_1d((x, y)) 将 (x, y) 作为两个单独的输入 x 和 y

个人感觉 torch 这种处理方式是有歧义的,如 torch.atleast_1d((x, y)) 和 torch.atleast_1d(x, y) 效果是一样的:

In [4]: y = torch.tensor(0.3)

In [5]: x = torch.tensor(0.3)

In [6]: torch.atleast_1d(x, y)
Out[6]: (tensor([0.3000]), tensor([0.3000]))

In [7]: torch.atleast_1d((x, y))
Out[7]: (tensor([0.3000]), tensor([0.3000]))

所以,根据本次设计接口的签名: paddle.atleast_1d(*inputs),将 paddle.atleast_1d((x, y)) 作为一个输入,而 paddle.atleast_1d(x, y) 是两个输入。

请悉知 ~

@megemini
Copy link
Contributor Author

Update 20231115

  • 将方法加入到 tensor_method_func
  • 增加相应单测

Copy link
Contributor

@jeff41404 jeff41404 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@ooooo-create ooooo-create left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luotao1
Copy link
Contributor

luotao1 commented Nov 16, 2023

@megemini 请补充对应的中文文档

Copy link
Contributor

@sunzhongkai588 sunzhongkai588 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM~

@luotao1 luotao1 changed the title 【Hackathon 5th No.33】为 Paddle 新增 atleast_1d / atleast_2d / atleast_3d API 【Hackathon 5th No.33】为 Paddle 新增 atleast_1d / atleast_2d / atleast_3d API -part Nov 16, 2023
@luotao1 luotao1 merged commit 557499b into PaddlePaddle:develop Nov 16, 2023
28 checks passed
@luotao1
Copy link
Contributor

luotao1 commented Nov 17, 2023

@megemini 顺师傅能修一下单测随机挂么?

单测 : test_atleast_nd, PR: 57785,CI : PR-CI-Mac-Python3,url : https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/9541163
单测 : test_atleast_nd, PR: 59070,CI : PR-CI-Mac-Python3,url : https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/9541296
单测 : test_atleast_nd, PR: 57879,CI : PR-CI-Mac-Python3,url : https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/9541705
单测 : test_atleast_nd, PR: 59040,CI : PR-CI-Mac-Python3,url : https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/9541888

757d1e2d4f5583d91f23a1d1801f7e34

SecretXV pushed a commit to SecretXV/Paddle that referenced this pull request Nov 28, 2023
… API -part (PaddlePaddle#58323)

* [Init] add atleast api

* [Add] add atleast test

* [Fix] import atleast

* [Change] test_atleast.py to test_atleast_nd.py and add bool data type test

* [Update] update dtype supports and unittest

* [Fix] dtype error unittest

* [Change] static test with test_with_pir_api

* [Add] atleast_Nd as tensor method
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants