[Hackathon 6th Code Camp No.15] support earthformer #816
Conversation
|
Thanks for your contribution! |
|
Yang-Changhui seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
|
❌ The PR is not created using PR's template. You can refer to this Demo. |
|
得先签署一下cla协议， |
|
这边显示已经同意啦 |
|
|
||
| USE_SAMPLED_DATA: false | ||
| # set train and evaluate data path | ||
| FILE_PATH: /home/aistudio/data/data260191/enso_round1_train_20210201 |
|
|
||
|
|
||
|
|
||
|
|
| max_batch_size: 16 | ||
| num_cpu_threads: 4 | ||
| batch_size: 1 | ||
| data_path: /home/aistudio/data/data260191/enso_round1_train_20210201/SODA_train.nc |
| batch_size: 16 | ||
|
|
||
| INFER: | ||
| pretrained_model_path: /home/aistudio/best_model.pdparams |
ppsci/arch/cuboid_transformer.py
Outdated
| from functools import lru_cache | ||
| from typing import Tuple | ||
| from collections import OrderedDict | ||
| from .cuboid_transformer_patterns import CuboidSelfAttentionPatterns, CuboidCrossAttentionPatterns |
There was a problem hiding this comment.
建议使用绝对路径导入，ppsci.xxx
ppsci/arch/cuboid_transformer.py
Outdated
| def get_activation(act, inplace=False, **kwargs): | ||
| """ | ||
|
|
||
| Parameters | ||
| ---------- | ||
| act | ||
| Name of the activation | ||
| inplace | ||
| Whether to perform inplace activation | ||
|
|
||
| Returns | ||
| ------- | ||
| activation_layer | ||
| The activation | ||
| """ | ||
| if act is None: | ||
| return lambda x: x | ||
| if isinstance(act, str): | ||
| if act == 'leaky': | ||
| negative_slope = kwargs.get('negative_slope', 0.1) | ||
| return paddle.nn.LeakyReLU(negative_slope=negative_slope) | ||
| elif act == 'identity': | ||
| return paddle.nn.Identity() | ||
| elif act == 'elu': | ||
| return paddle.nn.ELU() | ||
| elif act == 'gelu': | ||
| return paddle.nn.GELU() | ||
| elif act == 'relu': | ||
| return paddle.nn.ReLU() | ||
| elif act == 'sigmoid': | ||
| return paddle.nn.Sigmoid() | ||
| elif act == 'tanh': | ||
| return paddle.nn.Tanh() | ||
| elif act == 'softrelu' or act == 'softplus': | ||
| return paddle.nn.Softplus() | ||
| elif act == 'softsign': | ||
| return paddle.nn.Softsign() | ||
| else: | ||
| raise NotImplementedError( | ||
| 'act="{}" is not supported. Try to include it if you can find that in ' | ||
| 'https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/nn/Overview_cn.html' | ||
| .format(act)) | ||
| else: | ||
| return act |
There was a problem hiding this comment.
activation.py里应该有类似函数
There was a problem hiding this comment.
这个函数是有的，但是其中使用的LeakyReLU中的negative_slope参数做了修改，还有激活函数不全，这种情况怎么做修改呢
ppsci/arch/cuboid_transformer.py
Outdated
|
|
||
|
|
||
| class RMSNorm(paddle.nn.Layer): | ||
|
|
ppsci/arch/cuboid_transformer.py
Outdated
| out_4 = paddle.create_parameter( | ||
| shape=init_data.shape, | ||
| dtype=init_data.dtype, | ||
| default_initializer=nn.initializer.Assign(init_data)) | ||
| out_4.stop_gradient = not True | ||
| self.scale = out_4 | ||
| self.add_parameter(name='scale', parameter=self.scale) | ||
| if self.bias: | ||
| init_data = paddle.zeros(d) | ||
| out_5 = paddle.create_parameter( | ||
| shape=init_data.shape, | ||
| dtype=init_data.dtype, | ||
| default_initializer=nn.initializer.Assign(init_data)) | ||
| out_5.stop_gradient = not True | ||
| self.offset = out_5 | ||
| self.add_parameter(name='offset', parameter=self.offset) |
There was a problem hiding this comment.
转换后的变量名改一下，不要使用意义不明的命名
ppsci/data/dataset/enso_dataset.py
Outdated
|
|
||
|
|
||
|
|
||
|
|
||
|
|
|
代码commit前请安装pre-commit |
docs/zh/api/arch.md
Outdated
| - HEDeepONets | ||
| - ChipDeepONets | ||
| - AutoEncoder | ||
| - CuboidTransformerModel |
There was a problem hiding this comment.
CuboidTransformerModel建议改为CuboidTransformer
| input_shape: [12, 24, 48, 1] | ||
| target_shape: [14, 24, 48, 1] | ||
| base_units: 64 | ||
| # block_units: null |
There was a problem hiding this comment.
block_units这个参数是需要的吗，不需要的话可以删除？
examples/earthformer/enso/train.py
Outdated
| num_blocks = len(cfg.MODEL.afno["enc_depth"]) | ||
| if isinstance(cfg.MODEL["self_pattern"], str): | ||
| enc_attn_patterns = [cfg.MODEL["self_pattern"]] * num_blocks | ||
|
|
||
| if isinstance(cfg.MODEL["cross_self_pattern"], str): | ||
| dec_self_attn_patterns = [cfg.MODEL["cross_self_pattern"]] * num_blocks | ||
|
|
||
| if isinstance(cfg.MODEL["cross_pattern"], str): | ||
| dec_cross_attn_patterns = [cfg.MODEL["cross_pattern"]] * num_blocks |
There was a problem hiding this comment.
这段逻辑只受到num_blocks的控制，是否能给CuboidTransformer增加一个num_blocks的参数，然后这段配置生成逻辑放到CuboidTransformer.__init__里去呢？我觉得这样的做法更合理
可以参考hidden_size的写法：
examples/earthformer/enso/train.py
Outdated
|
|
||
| # # init optimizer and lr scheduler | ||
| lr_scheduler_cfg = dict(cfg.TRAIN.lr_scheduler) | ||
| lr_scheduler_cfg.update({"iters_per_epoch": ITERS_PER_EPOCH}) |
There was a problem hiding this comment.
把这行移动到eta_min=xxx的上面也可以把，不需要转dict再update了？
| { | ||
| "params": [p for n, p in model.named_parameters() if n in decay_parameters], | ||
| "weight_decay": cfg.TRAIN.wd, | ||
| }, |
There was a problem hiding this comment.
这里是否有必要为非decay_parameters设置wd呢？因为Optimizer的实例化中有一个同样的设置：
There was a problem hiding this comment.
请问，weight_decay这个参数的初始化中是0.01，cfg.TRAIN.wd为1e-5，Optimizer的实例化是按照那个初始化呢
There was a problem hiding this comment.
请问，weight_decay这个参数的初始化中是0.01，cfg.TRAIN.wd为1e-5，Optimizer的实例化是按照那个初始化呢
看了下官网的，可以先这么写吧，问题不大。
There was a problem hiding this comment.
我尝试把这个参数去掉，重新训练了一下，对结果没什么影响，感觉可以去掉
There was a problem hiding this comment.
我尝试把这个参数去掉，重新训练了一下，对结果没什么影响，感觉可以去掉
确认了下外部的wd是全局指定，并且可以被params参数内指定的wd覆盖，
而learning_rate是缩放倍率的关系，即最终的lr是全局lr乘以params字典内指定的lr
所以删掉跟全局wd重复的字段应该是没问题。
上述行为跟torch应该是一致的。
There was a problem hiding this comment.
不建议引入registry机制，会增加代码阅读难度和调试难度，尤其是可能会影响python报错栈
ppsci/arch/cuboid_transformer.py
Outdated
| def get_activation(act, inplace=False, **kwargs): | ||
| """ | ||
| Parameters | ||
| ---------- | ||
| act | ||
| Name of the activation | ||
| inplace | ||
| Whether to perform inplace activation | ||
|
|
||
| Returns | ||
| ------- | ||
| activation_layer | ||
| The activation | ||
| """ | ||
| if act is None: | ||
| return lambda x: x | ||
| if isinstance(act, str): | ||
| if act == "leaky": | ||
| negative_slope = kwargs.get("negative_slope", 0.1) | ||
| return paddle.nn.LeakyReLU(negative_slope=negative_slope) | ||
| elif act == "identity": | ||
| return paddle.nn.Identity() | ||
| elif act == "elu": | ||
| return paddle.nn.ELU() | ||
| elif act == "gelu": | ||
| return paddle.nn.GELU() | ||
| elif act == "relu": | ||
| return paddle.nn.ReLU() | ||
| elif act == "sigmoid": | ||
| return paddle.nn.Sigmoid() | ||
| elif act == "tanh": | ||
| return paddle.nn.Tanh() | ||
| elif act == "softrelu" or act == "softplus": | ||
| return paddle.nn.Softplus() | ||
| elif act == "softsign": | ||
| return paddle.nn.Softsign() | ||
| else: | ||
| raise NotImplementedError( | ||
| 'act="{}" is not supported. Try to include it if you can find that in ' | ||
| "https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/nn/Overview_cn.html".format( | ||
| act | ||
| ) | ||
| ) | ||
| else: | ||
| return act |
There was a problem hiding this comment.
建议完善下ppsci/arch/activation.py，尽量复用已有代码
ppsci/arch/cuboid_transformer.py
Outdated
| whether use bias term for RMSNorm, disabled by | ||
| default because RMSNorm doesn't enforce re-centering invariance. | ||
| """ | ||
| super(RMSNorm, self).__init__() |
There was a problem hiding this comment.
改为super().__init__()
ppsci/arch/cuboid_transformer.py
Outdated
| self.scale = paddle.create_parameter( | ||
| shape=init_data.shape, | ||
| dtype=init_data.dtype, | ||
| default_initializer=nn.initializer.Assign(init_data), |
There was a problem hiding this comment.
可以直接使用nn.initializer.Constant(1.0)
ppsci/arch/cuboid_transformer.py
Outdated
| init_data = paddle.ones(d) | ||
| self.scale = paddle.create_parameter( | ||
| shape=init_data.shape, | ||
| dtype=init_data.dtype, | ||
| default_initializer=nn.initializer.Assign(init_data), | ||
| ) | ||
| self.scale.stop_gradient = not True | ||
| self.add_parameter(name="scale", parameter=self.scale) | ||
| if self.bias: | ||
| init_data = paddle.zeros(d) | ||
| self.offset = paddle.create_parameter( | ||
| shape=init_data.shape, | ||
| dtype=init_data.dtype, | ||
| default_initializer=nn.initializer.Assign(init_data), | ||
| ) | ||
| self.offset.stop_gradient = not True | ||
| self.add_parameter(name="offset", parameter=self.offset) |
There was a problem hiding this comment.
可以直接使用nn.initializer.Constant(value)代替Assign初始化，减少代码量
… into add_earthformer
…ddleScience into add_earthformer
ppsci/arch/cuboid_transformer.py
Outdated
| import paddle | ||
| import paddle.nn.functional as F | ||
| from paddle import nn | ||
| from paddle.distributed.fleet.utils import recompute |
There was a problem hiding this comment.
建议使用fleet.utils.recompute的方式调用
| { | ||
| "params": [p for n, p in model.named_parameters() if n in decay_parameters], | ||
| "weight_decay": cfg.TRAIN.wd, | ||
| }, |
There was a problem hiding this comment.
请问，weight_decay这个参数的初始化中是0.01，cfg.TRAIN.wd为1e-5，Optimizer的实例化是按照那个初始化呢
看了下官网的，可以先这么写吧，问题不大。
|
|
||
| input_spec = [ | ||
| { | ||
| key: InputSpec([1, 12, 24, 48, 1], "float32", name=key) |
There was a problem hiding this comment.
第一个维度是否可以改为None？
#816 (comment) 因为reshape操作，设置为None会导致维度冲突
哦哦好的，那先写成1好了
| optimizer_grouped_parameters = [ | ||
| { | ||
| "params": [p for n, p in model.named_parameters() if n in decay_parameters], | ||
| "weight_decay": cfg.TRAIN.wd, |
examples/earthformer/sevir/train.py
Outdated
| }, | ||
| ] | ||
|
|
||
| # # init optimizer and lr scheduler |
ppsci/data/dataset/enso_dataset.py
Outdated
| assert self.target_nino.shape[1] == self.out_len - NINO_WINDOW_T + 1 | ||
| return self.sst, self.target_nino | ||
|
|
||
| def GetDataShape(self): |
There was a problem hiding this comment.
对于入参的合法性检查，建议使用if+raise XXXError判断，assert断言用于内部产生的变量判断是否符合预期，两种方法使用场景是不同的
There was a problem hiding this comment.
get_parameter_names是一个高度定制化并不是很通用的函数，不建议加到utils里去
ppsci/utils/get_param.py
Outdated
There was a problem hiding this comment.
该函数可以与调用该函数的代码放在一起，但不建议单独建一个文件，不太有必要。
examples/earthformer/enso/train.py
Outdated
| @@ -0,0 +1,261 @@ | |||
| import enso_metric | |||
There was a problem hiding this comment.
本地python模块的import顺序应该是在最下方跟ppsci一起才对，为什么会跑到上面呢？
examples/earthformer/enso/train.py
Outdated
| import hydra | ||
| import numpy as np | ||
| import paddle | ||
| import xarray as xr |
There was a problem hiding this comment.
对于这类不通用只为某几个案例使用的模块导入行为，可以分两种情况考虑
- 不会被ppsci模块导入，如train.py，可以使用以下语句代替，能提示用户如何安装该模块，保证用户只有在运行该文件时才报错。
try:
import xxx
except ModuleNotFoundError:
raise ModuleNotFoundError(
"Please install xxx with `pip install xxx`."
)- 对于会被ppsci模块导入的子模块，如enso_dataset.py，作如下修改，保证用户只有在使用该类时才报错
impot
try:
import xxx
except ModuleNotFoundError:
pass # 这样不影响正常用户使用
Class XXXDataset:
def __init__(....):
super().__init__()
if importlib.util.find_spec("xxx") is None:
raise ModuleNotFoundError(
"To use RadarDataset, please install 'xxx' via: `pip install "
"xxx` first."
) # 在实例化该Dataset的时候检查并提示安装命令否则随着代码变多，项目的不必要依赖会越来越多，这显然是不合理的
|
建议重新进行一次完整的pre-commit，code-style-check流水线挂了 |
| cross_last_n_frames (int, optional): The cross_last_n_frames of decoder. Defaults to None. | ||
| qkv_bias (bool, optional): Whether to enable bias in calculating qkv attention. Defaults to False. | ||
| qk_scale (float, optional): Whether to enable scale factor when calculating the attention. Defaults to None. | ||
| attn_drop (float, optional): The attention dropout.. Defaults to 0.0. |
| qkv_bias (bool, optional): Whether to enable bias in calculating qkv attention. Defaults to False. | ||
| qk_scale (float, optional): Whether to enable scale factor when calculating the attention. Defaults to None. | ||
| attn_drop (float, optional): The attention dropout.. Defaults to 0.0. | ||
| proj_drop (float, optional): The projrction dropout.. Defaults to 0.0. |
| max_temporal_relative (int, optional): The max temporal. Defaults to 50. | ||
| norm_layer (str, optional): The normalization layer. Defaults to "layer_norm". | ||
| use_global_vector (bool, optional): Whether to use the global vector or not. Defaults to True. | ||
| separate_global_qkv (bool, optional): Whether to use different network to calc q_global, k_global, v_global. . Defaults to False. |
There was a problem hiding this comment.
删除v_global后多余的标点符号
… into add_earthformer
…ddleScience into add_earthformer
| normalization: str = "layer_norm", | ||
| layer_norm_eps: float = 1e-05, | ||
| pre_norm: bool = False, | ||
| linear_init_mode="0", |
There was a problem hiding this comment.
跟上面一样，完善数据类型，其他关键的类也需要完善下
linear_init_mode: str="0",
norm_init_mode: str="0",
| def compute_enso_score( | ||
| y_pred, y_true, acc_weight: Optional[Union[str, np.ndarray, paddle.Tensor]] = None | ||
| ): | ||
| """_summary_ |
| Args: | ||
| y_pred (paddle.Tensor): predict data | ||
| y_true (paddle.Tensor): true data | ||
| acc_weight (Optional[Union[str, np.ndarray, paddle.Tensor]], optional): _description_. Defaults to None.use default acc_weight specified at https://tianchi.aliyun.com/competition/entrance/531871/information |
| Returns: | ||
| acc: | ||
| rmse: |
|
|
||
|
|
||
| def sst_to_nino(sst: paddle.Tensor, normalize_sst: bool = True, detach: bool = True): | ||
| """_summary_ |
| Args: | ||
| sst (paddle.Tensor): Shape = (N, T, H, W) | ||
| normalize_sst (bool, optional): Defaults to True. | ||
| detach (bool, optional): Defaults to True. |
| Raises: | ||
| NotImplementedError: _description_ | ||
| """ |
| dtype=init_data.dtype, | ||
| default_initializer=nn.initializer.Constant(1.0), | ||
| ) | ||
| self.scale.stop_gradient = not True |
There was a problem hiding this comment.
not True 改为 False，其他类似代码也替换下
| from ppsci.data.dataset.csv_dataset import CSVDataset | ||
| from ppsci.data.dataset.csv_dataset import IterableCSVDataset | ||
| from ppsci.data.dataset.cylinder_dataset import MeshCylinderDataset | ||
| from ppsci.data.dataset.dgmr_dataset import DGMRDataset |
|
@HydrogenSulfate 你好，我现在有一个简单的前向，initencoder，相同的网络，相同输入，相同权重，使用reprod_log检查，误差为0.00067，这是最小测试单元， |
5 participants
PR types
New features
PR changes
Others
Describe
增加EarthFormer模型