[GPU] LSTMSequence and LSTMCell optimization #26767

michal-miotk · 2024-09-24T14:09:12Z

Details:

creating simple primitive for lstm_sequence to be faster than previous approach using many primitives
using oneDNN
based on commit c99ddc0 from 25732

Tickets:

146601

commit 232d272f11fbe65e82fa9787260a8b9d34b57d20 Author: michal-miotk <michal.miotk@intel.com> Date: Mon Jul 29 11:17:47 2024 +0000 wip commit e642ca3 Author: michal-miotk <michal.miotk@intel.com> Date: Sun Jul 28 22:08:24 2024 +0000 wip commit c6b74d3 Author: michal-miotk <michal.miotk@intel.com> Date: Fri Jul 26 14:10:26 2024 +0000 wip commit 0451429 Author: michal-miotk <michal.miotk@intel.com> Date: Thu Jul 25 20:35:11 2024 +0000 wip3

commit 1164592 Author: michal-miotk <michal.miotk@intel.com> Date: Tue Aug 6 09:25:45 2024 +0000 wip commit 8b2c049 Author: michal-miotk <michal.miotk@intel.com> Date: Tue Aug 6 09:24:02 2024 +0000 wip commit 886b412 Author: michal-miotk <michal.miotk@intel.com> Date: Mon Aug 5 14:59:14 2024 +0000 wip commit 08fb207 Author: michal-miotk <michal.miotk@intel.com> Date: Sun Aug 4 20:21:38 2024 +0000 wip, errors on half commit 125884d Author: michal-miotk <michal.miotk@intel.com> Date: Sat Aug 3 23:59:58 2024 +0000 wip commit af4f209 Author: michal-miotk <michal.miotk@intel.com> Date: Fri Aug 2 17:58:38 2024 +0000 wip commit 12626fc Author: michal-miotk <michal.miotk@intel.com> Date: Fri Aug 2 10:52:15 2024 +0000 wip commit dfdd052 Author: michal-miotk <michal.miotk@intel.com> Date: Thu Aug 1 15:38:41 2024 +0000 wip commit 54ee912 Author: michal-miotk <michal.miotk@intel.com> Date: Thu Aug 1 11:01:55 2024 +0000 only bfyx layout commit 240fe4a Author: michal-miotk <michal.miotk@intel.com> Date: Thu Aug 1 10:34:45 2024 +0000 two outputs from prim commit bc775be Author: michal-miotk <michal.miotk@intel.com> Date: Wed Jul 31 22:13:14 2024 +0000 wip commit d1cfd60 Author: michal-miotk <michal.miotk@intel.com> Date: Wed Jul 31 22:07:06 2024 +0000 wip commit 7d18884 Author: michal-miotk <michal.miotk@intel.com> Date: Wed Jul 31 19:19:04 2024 +0000 begin of handling reverse commit 39f64af Author: michal-miotk <michal.miotk@intel.com> Date: Wed Jul 31 15:37:06 2024 +0000 betterbetter commit 67b3c9a Author: michal-miotk <michal.miotk@intel.com> Date: Wed Jul 31 13:12:39 2024 +0000 better commit 6ded5aa Author: michal-miotk <michal.miotk@intel.com> Date: Wed Jul 31 10:12:31 2024 +0000 wip commit 1ccdacc Author: michal-miotk <michal.miotk@intel.com> Date: Tue Jul 30 23:07:21 2024 +0000 wip commit ab1307c Author: michal-miotk <michal.miotk@intel.com> Date: Tue Jul 30 22:00:50 2024 +0000 test passed commit bc65969 Author: michal-miotk <michal.miotk@intel.com> Date: Tue Jul 30 15:37:20 2024 +0000 wip commit 03cbf57 Author: michal-miotk <michal.miotk@intel.com> Date: Tue Jul 30 15:15:06 2024 +0000 only 2 outputs commit fd5f3dc Author: michal-miotk <michal.miotk@intel.com> Date: Tue Jul 30 14:47:12 2024 +0000 wip commit 939d23c Author: michal-miotk <michal.miotk@intel.com> Date: Tue Jul 30 11:34:56 2024 +0000 wip commit 2bb561f Author: michal-miotk <michal.miotk@intel.com> Date: Tue Jul 30 09:28:03 2024 +0000 added to binary buffer commit 1ef83ff Author: michal-miotk <michal.miotk@intel.com> Date: Mon Jul 29 22:30:57 2024 +0000 not works

…tion only in gpu plugin

…weights pass

This reverts commit 165dd9b.

Lyamin-Roman · 2024-10-18T14:54:56Z

src/plugins/intel_gpu/include/intel_gpu/graph/program.hpp

@@ -45,6 +45,7 @@ struct program {
    friend class reorder_inputs;                    // to be removed when possible
    friend class remove_redundant_reorders;         // to be removed when possible
    friend class post_optimize_weights;             // to be removed when possible
+    friend class post_optimize_lstm_weights_and_output;             // to be removed when possible


nit: alignment

Lyamin-Roman · 2024-10-18T14:56:04Z

src/plugins/intel_gpu/include/intel_gpu/primitives/rnn.hpp

+struct RNNParams{
+    RNNParams() = default;
+    RNNParams(const RNNParams&) = default;
+    RNNParams(const primitive_id& id,


Why isn't primitive_base used as the base class?

Lyamin-Roman · 2024-10-18T14:57:45Z

src/plugins/intel_gpu/include/intel_gpu/primitives/rnn.hpp

+    primitive_id id;
+    input_info x;
+    input_info initial_hidden_state;
+    input_info initial_cell_state;/// @brief for lstm_elt primitive field for cell input_info


nit: please, fix @brief location

Lyamin-Roman · 2024-10-18T15:14:00Z

src/plugins/intel_gpu/src/graph/graph_optimizer/post_optimize_lstm_weights_and_output.cpp

+
+// function which prepares given primitive for weights optimization
+template<typename T>
+void post_optimize_lstm_weights_and_output::optimize_lstm_weights(T& node, program& p) {


Couldn't this be implemented as part of post_optimize_weights::optimize_weights?
It looks like a lot of code should match

Lyamin-Roman · 2024-10-18T15:22:33Z

src/plugins/intel_gpu/src/graph/impls/ocl/lstm_cell.cpp

+        if (!primitive->params.activations.empty()) {
+            auto a_sz = primitive->params.activations.size();
+            auto param_sz = primitive->params.activation_params.size();
+            OPENVINO_ASSERT(param_sz == 0|| a_sz == param_sz, "[GPU] Unexpected activation params count in lstm_cell impl: ", param_sz);


nit:

Suggested change

OPENVINO_ASSERT(param_sz == 0|| a_sz == param_sz, "[GPU] Unexpected activation params count in lstm_cell impl: ", param_sz);

OPENVINO_ASSERT(param_sz == 0 || a_sz == param_sz, "[GPU] Unexpected activation params count in lstm_cell impl: ", param_sz);

Lyamin-Roman · 2024-10-18T17:12:41Z

src/plugins/intel_gpu/src/graph/lstm_cell.cpp

+    return {cldnn::layout{ShapeType{lstm_batch_size, lstm_hidden_size}, input_layout_x.data_type, input_layout_x.format}, \
+            cldnn::layout{ShapeType{lstm_batch_size, lstm_hidden_size}, input_layout_x.data_type, input_layout_x.format}};


Suggested change

return {cldnn::layout{ShapeType{lstm_batch_size, lstm_hidden_size}, input_layout_x.data_type, input_layout_x.format}, \

cldnn::layout{ShapeType{lstm_batch_size, lstm_hidden_size}, input_layout_x.data_type, input_layout_x.format}};

auto out_layout = cldnn::layout{ShapeType{lstm_batch_size, lstm_hidden_size}, input_layout_x.data_type, input_layout_x.format};

return {out_layout, out_layout };

Lyamin-Roman · 2024-10-18T17:22:39Z

src/plugins/intel_gpu/src/kernel_selector/cl_kernels/lstm_cell_and_seq_bfyx.cl

+                        #ifdef SEQUENCE
+                            INPUT1_TYPE_VEC initial_block = READ_VEC(0, &initial_hidden_state[INPUT1_GET_INDEX(b, 0, j*VEC_SIZE, 0)]);
+                            INPUT4_TYPE_VEC r_block = READ_VEC(0, &R[INPUT4_GET_INDEX(0, weight_idx, j*VEC_SIZE, 0)]);
+                            hidden_result += initial_block.s0*r_block.s0 + initial_block.s1*r_block.s1 + initial_block.s2*r_block.s2 + initial_block.s3*r_block.s3 + initial_block.s4*r_block.s4 + initial_block.s5*r_block.s5 + initial_block.s6*r_block.s6 + initial_block.s7*r_block.s7;


nit: Maybe some macro can reduce the length of a line of similar operations

Lyamin-Roman · 2024-10-18T17:32:12Z

src/plugins/intel_gpu/src/kernel_selector/kernels/lstm/lstm_cell_kernel_ref.cpp

+    k.EnableTensorOffset();
+    k.EnableTensorPitches();
+    k.EnableBatching();
+    k.EnableDynamicShapesSupport();


Is there support for dynamic shapes?

Lyamin-Roman · 2024-10-18T17:34:59Z

src/plugins/intel_gpu/src/plugin/ops/rnn.cpp

-
+    auto inputs = p.GetInputInfo(op);
+    /*
+    if (op->get_input_shape(2).size() != 2 || op->get_input_shape(3).size() != 2 \


Is this check no longer valid?

Lyamin-Roman · 2024-10-18T17:35:45Z

src/plugins/intel_gpu/src/plugin/ops/rnn.cpp

+         cldnn::input_info(), "", "", clip, activations, \
+                                            activation_params, cldnn::lstm_weights_order::fizo, direction, cldnn::padding(), \
+            static_cast<int>(op->get_output_size())}, 0);
+        //prim.output_data_types = get_output_data_types(op);


Commented code

This reverts commit 6af1f3f.

This reverts commit f8dbec3.

…ptimize_weights pass" This reverts commit 00826ad.

michal-miotk added 30 commits July 18, 2024 11:36

compiles lstm_seq

9ce143a

more kernel args

027f991

bigger proper run chances

c191c58

19jul

d461e66

inference works

01fa2ac

in middle of implementation

1f017fd

problems with inputs get element in kernel

5787c7d

not compile

837db22

wipx

d4ce531

wip

19c268e

solved problem with too much inputs kernel

f5273bc

wip

d50b3be

more changes

63a8dfd

wip

f54ecc1

wip

3748a11

wip

fae772a

proper shape for 2 outputs

c00ff8a

cleaning

31fcb79

Merge branch 'master' into lstm2

4b16eef

updated to new primitive_base api, disabled lstm to tensor transforma…

dcad182

…tion only in gpu plugin

now it should compile on windows, changed kernel name

d6aeb54

deleted cell, deleted input_forget

9688f63

generic primitive

5003d47

fix compilation problem, smaller lws

5937b14

wip

8b31a91

wip, not resolved fail on dynamic

2ff5a7c

fixed failing dynamic test

2d9e5c6

change name cldnn::rnn -> cldnn::lstm_seq

702e941

github-actions bot added the category: IE Tests OpenVINO Test: plugins and common label Oct 14, 2024

michal-miotk added 18 commits October 14, 2024 15:24

Merge branch 'master' into lstm_with_onednn

d0fb8b4

initialized memory in infer_request_dynamic tests

a1497c4

fix for failing caching tests

f12aebd

deleted event handling as in case in in_order que it is not used

6170710

preventing duplicates

01dc7dc

repairs in infer_request set and get tensor

e9bf370

fused test repair

7158776

set in order queue as default test config

5e21106

only bfyx format for lstm_seq

daa83b5

skipping conv fusion tests

6af1f3f

skipping f16 deconv gpu tests

02942e5

conv_fp32_multi_eltquant skip in conv_fusion_test

f8dbec3

Merge branch 'master' into lstm_with_onednn

2abe8f8

update hash as input format of weights is custom after post_optimize_…

00826ad

…weights pass

change format in conv_fp32_multi_eltwise_concat basic test

36b4853

fix shape calc for onednn, only bfyx supported for lstmocl

c358ab3

Revert "optimizations for bigger gpus"

19b1d93

This reverts commit 165dd9b.

deleted all get_index safe in lstm bfyx kernel

4da2df6

Lyamin-Roman reviewed Oct 18, 2024

View reviewed changes

michal-miotk added 10 commits October 18, 2024 22:09

applying review part1

303bf7d

fix check of dimensions

bf9f13f

fix check of input dim lstm cell

459e1ad

enable onednn for tests ON, LSTMSeq accept bfyx and fbyx format

14e53f4

dot op, vec_size=4

063ac02

Revert "skipping conv fusion tests"

892131b

This reverts commit 6af1f3f.

Revert "conv_fp32_multi_eltquant skip in conv_fusion_test"

b539a3f

This reverts commit f8dbec3.

lstm_weights optimization is part of post_optimize_weights

dc8ac73

fix forbiddnen size_t->int conversion

a5165a8

Revert "update hash as input format of weights is custom after post_o…

cc6b4b5

…ptimize_weights pass" This reverts commit 00826ad.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GPU] LSTMSequence and LSTMCell optimization #26767

[GPU] LSTMSequence and LSTMCell optimization #26767

michal-miotk commented Sep 24, 2024 •

edited

Loading

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 18, 2024

Lyamin-Roman Oct 18, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 18, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 20, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 18, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 18, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 20, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 18, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 18, 2024

Lyamin-Roman Oct 18, 2024

michal-miotk Oct 18, 2024

	OPENVINO_ASSERT(param_sz == 0\|\| a_sz == param_sz, "[GPU] Unexpected activation params count in lstm_cell impl: ", param_sz);
	OPENVINO_ASSERT(param_sz == 0 \|\| a_sz == param_sz, "[GPU] Unexpected activation params count in lstm_cell impl: ", param_sz);

		return {cldnn::layout{ShapeType{lstm_batch_size, lstm_hidden_size}, input_layout_x.data_type, input_layout_x.format}, \
		cldnn::layout{ShapeType{lstm_batch_size, lstm_hidden_size}, input_layout_x.data_type, input_layout_x.format}};

[GPU] LSTMSequence and LSTMCell optimization #26767

Are you sure you want to change the base?

[GPU] LSTMSequence and LSTMCell optimization #26767

Conversation

michal-miotk commented Sep 24, 2024 • edited Loading

Details:

Tickets:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michal-miotk commented Sep 24, 2024 •

edited

Loading