converting SD 2.1 models with onnx-fp16 optimization leads to black images #364
Description
As noted in the ORT docs, https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/python/tools/transformers/models/stable_diffusion#optimize-onnx-pipeline, converting an SD v2.1-based model with the ONNX fp16 optimization on will not work correctly and all outputs will be black images.
With the recommended flags, this fails with a type error:
[2023-04-30 13:56:59,958] DEBUG: 3452945 140182939832320 onnx_web.convert.diffusion.diffusers: UNET config: FrozenDict([('sample_size', 96), ('in_channels', 4), ('out_channels', 4), ('center_input_sample', False)
, ('flip_sin_to_cos', True), ('freq_shift', 0), ('down_block_types', ('CrossAttnDownBlock2D', 'CrossAttnDownBlock2D', 'CrossAttnDownBlock2D', 'DownBlock2D')), ('mid_block_type', 'UNetMidBlock2DCrossAttn'), ('up_b
lock_types', ('UpBlock2D', 'CrossAttnUpBlock2D', 'CrossAttnUpBlock2D', 'CrossAttnUpBlock2D')), ('only_cross_attention', False), ('block_out_channels', (320, 640, 1280, 1280)), ('layers_per_block', 2), ('downsampl
e_padding', 1), ('mid_block_scale_factor', 1), ('act_fn', 'silu'), ('norm_num_groups', 32), ('norm_eps', 1e-05), ('cross_attention_dim', 1024), ('encoder_hid_dim', None), ('attention_head_dim', [5, 10, 20, 20]),
('dual_cross_attention', False), ('use_linear_projection', True), ('class_embed_type', None), ('addition_embed_type', None), ('num_class_embeds', None), ('upcast_attention', True), ('resnet_time_scale_shift', 'de
fault'), ('resnet_skip_time_act', False), ('resnet_out_scale_factor', 1.0), ('time_embedding_type', 'positional'), ('time_embedding_dim', None), ('time_embedding_act_fn', None), ('timestep_post_act', None), ('tim
e_cond_proj_dim', None), ('conv_in_kernel', 3), ('conv_out_kernel', 3), ('projection_class_embeddings_input_dim', None), ('class_embeddings_concat', False), ('mid_block_only_cross_attention', None), ('cross_atten
tion_norm', None), ('addition_embed_type_num_heads', 64)])
[2023-04-30 13:57:23,362] INFO: 3452945 140182939832320 onnx_web.convert.utils: converting model to fp16 internally: /opt/onnx-web/api/../models/diffusion-sticker-art/unet/model.onnx
[2023-04-30 13:57:24,600] DEBUG: 3452945 140182939832320 onnxruntime.transformers.float16: fp16 parameters: min_positive_val=5.96e-08 max_finite_val=65504.0 keep_io_types=True disable_shape_infer=True op_block_li
st={'Attention', 'MultiHeadAttention'} node_block_list=set() force_fp16_initializers=False
The config attributes {'encoder_hid_dim': None, 'addition_embed_type': None, 'resnet_skip_time_act': False, 'resnet_out_scale_factor': 1.0, 'time_embedding_dim': None, 'time_embedding_act_fn': None, 'class_embedd
ings_concat': False, 'mid_block_only_cross_attention': None, 'cross_attention_norm': None, 'addition_embed_type_num_heads': 64} were passed to UNet2DConditionModel_CNet, but are not expected and will be ignored.
Please verify your config.json configuration file.
[2023-04-30 13:58:02,505] INFO: 3452945 140182939832320 onnx_web.convert.utils: converting model to fp16 internally: /opt/onnx-web/api/../models/diffusion-sticker-art/cnet/model.onnx
[2023-04-30 13:58:03,781] DEBUG: 3452945 140182939832320 onnxruntime.transformers.float16: fp16 parameters: min_positive_val=5.96e-08 max_finite_val=65504.0 keep_io_types=True disable_shape_infer=True op_block_li
st={'Attention', 'MultiHeadAttention'} node_block_list=set() force_fp16_initializers=False
[2023-04-30 13:58:18,728] INFO: 3452945 140182939832320 onnx_web.convert.utils: converting model to fp16 internally: /opt/onnx-web/api/../models/diffusion-sticker-art/vae_decoder/model.onnx
[2023-04-30 13:58:19,474] DEBUG: 3452945 140182939832320 onnxruntime.transformers.float16: fp16 parameters: min_positive_val=5.96e-08 max_finite_val=65504.0 keep_io_types=True disable_shape_infer=True op_block_li
st={'DictVectorizer', 'CastMap', 'FeatureVectorizer', 'Binarizer', 'RoiAlign', 'Min', 'Scaler', 'Normalizer', 'LinearClassifier', 'OneHotEncoder', 'Imputer', 'NonMaxSuppression', 'CategoryMapper', 'ZipMap', 'Resi
ze', 'SVMRegressor', 'SVMClassifier', 'CumSum', 'Max', 'TopK', 'LinearRegressor', 'ArrayFeatureExtractor', 'LabelEncoder', 'Range', 'Upsample', 'TreeEnsembleRegressor', 'TreeEnsembleClassifier'} node_block_list=s
et() force_fp16_initializers=False
[2023-04-30 13:58:21,120] ERROR: 3452945 140182939832320 __main__: error converting diffusion model diffusion-sticker-art
Traceback (most recent call last):
File "/opt/onnx-web/api/onnx_web/convert/__main__.py", line 356, in convert_models
converted, dest = convert_diffusion_diffusers(
File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/opt/onnx-web/api/onnx_web/convert/diffusion/diffusers.py", line 544, in convert_diffusion_diffusers
unet=OnnxRuntimeModel.from_pretrained(output_path / "unet"),
File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/diffusers/pipelines/onnx_utils.py", line 205, in from_pretrained
return cls._from_pretrained(
File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/diffusers/pipelines/onnx_utils.py", line 172, in _from_pretrained
model = OnnxRuntimeModel.load_model(
File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/diffusers/pipelines/onnx_utils.py", line 77, in load_model
return ort.InferenceSession(path, providers=[provider], sess_options=sess_options)
File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 360, in __init__
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 397, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidGraph: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from ../models/diffusion-sticker-art/unet/model.onnx failed:This is an invalid model. Type Error: Typ
e 'tensor(float16)' of input parameter (/up_blocks.0/upsamplers.0/Constant_output_0) of operator (Resize) in node (/up_blocks.0/upsamplers.0/Resize) is invalid.