[TRANSFORMATIONS][GPU] Add GroupNormalization fusion to common optimizations #28387

jhajducz · 2025-01-11T02:03:54Z

Details:

Added GroupNormalization fusion pass that can handle pattern observed in many customer models that were exported via ONNX in a way that uses InstanceNormalization as a proxy for GroupNormalization. It covers also more traditional cases without additional instance norm related parameters.
Per suggestion from @vladimir-paramuzov, for now enabled GroupNormalization fusion only for GPU plugin. Once it will be verified that it doesn't cause regressions in other backends, we can enable it for them as well.

Tickets:

160436

mlukasze · 2025-01-13T05:52:12Z

build_jenkins

t-jankowski

Lgtm regarding Core part.

src/common/transformations/tests/common_optimizations/group_normalization_fusion_tests.cpp

src/core/include/openvino/core/type/element_type.hpp

praasz · 2025-01-13T11:14:44Z

build_jenkins

...mmon/transformations/src/transformations/common_optimizations/group_normalization_fusion.cpp

src/plugins/intel_gpu/src/plugin/transformations_pipeline.cpp

.../transformations/include/transformations/common_optimizations/group_normalization_fusion.hpp

...mmon/transformations/src/transformations/common_optimizations/group_normalization_fusion.cpp

src/common/transformations/tests/common_optimizations/group_normalization_fusion_tests.cpp

...mmon/transformations/src/transformations/common_optimizations/group_normalization_fusion.cpp

src/common/transformations/tests/common_optimizations/group_normalization_fusion_tests.cpp

.../transformations/include/transformations/common_optimizations/group_normalization_fusion.hpp

vladimir-paramuzov · 2025-01-17T05:07:06Z

...mmon/transformations/src/transformations/common_optimizations/group_normalization_fusion.cpp

+        {has_real_not_quantized_type, has_at_least_2d_shape, ov::pass::pattern::has_static_dim(1)}));
+
+    auto pre_mvn_shape_const_m = ov::pass::pattern::wrap_type<ov::op::v0::Constant>(ov::pass::pattern::all_of(
+        {has_integral_type, ov::pass::pattern::rank_equals(1), ov::pass::pattern::has_static_dim(0)}));


Some of those checks may be redundant as an operation will fail on validation pass if parameters are incorrect. For instance, this has_integral_type is not needed as Reshape op can't be created with non-integral type of target pattern

Removed has_integral_type predicate (it was used only for making sure that T_SHAPE for Reshape and T_IND for MVN will be of integral type, which is indeed redundant)

...mmon/transformations/src/transformations/common_optimizations/group_normalization_fusion.cpp

vladimir-paramuzov · 2025-01-17T05:20:59Z

...mmon/transformations/src/transformations/common_optimizations/group_normalization_fusion.cpp

+        }
+
+        if (pattern_map.count(instance_norm_gamma_m) > 0) {
+            const auto& instance_norm_gamma = pattern_map.at(instance_norm_gamma_m);


nit: I think this instance/group norm prefix can be removed in most of the cases to make the names shorter and easier to read

Actually, in worst case pattern will have gamma and beta parameters for both instance norm and group norm, so I prefer to keep these prefixes to distinguish them

This reverts commit 217bcdc.

…GroupNormalizationFusion tests

…izationFusion tests

…lizationFusion tests

…sformations pipeline

…es in GroupNormalizationFusion pass

…usion pass

e-ddykim · 2025-01-20T12:37:22Z

I checked that it works fine for SD 1.5 on GPU.

jhajducz · 2025-01-20T15:23:44Z

I on the other hand may have found potential accuracy issue, let me investigate it.

github-actions bot added category: Core OpenVINO Core (aka ngraph) category: GPU OpenVINO GPU plugin category: transformations OpenVINO Runtime library - Transformations category: CPP API OpenVINO CPP API bindings labels Jan 11, 2025

sys-openvino-ci added the ExternalIntelPR External contributor from Intel label Jan 11, 2025

jhajducz force-pushed the private/jhajducz/gpu_gn_fusion branch from 3a67c81 to ba87e35 Compare January 12, 2025 17:37

jhajducz marked this pull request as ready for review January 13, 2025 00:38

jhajducz requested review from a team as code owners January 13, 2025 00:38

jhajducz requested review from itikhono and removed request for a team January 13, 2025 00:38

t-jankowski approved these changes Jan 13, 2025

View reviewed changes

src/common/transformations/tests/common_optimizations/group_normalization_fusion_tests.cpp Show resolved Hide resolved

jhajducz force-pushed the private/jhajducz/gpu_gn_fusion branch from ba87e35 to 3bd623d Compare January 13, 2025 10:49

praasz requested changes Jan 13, 2025

View reviewed changes

src/core/include/openvino/core/type/element_type.hpp Outdated Show resolved Hide resolved

jhajducz force-pushed the private/jhajducz/gpu_gn_fusion branch from 3bd623d to d69391b Compare January 14, 2025 13:05

github-actions bot removed category: Core OpenVINO Core (aka ngraph) category: CPP API OpenVINO CPP API bindings labels Jan 14, 2025

jhajducz requested a review from praasz January 14, 2025 15:39

jhajducz force-pushed the private/jhajducz/gpu_gn_fusion branch from d69391b to b1ac67d Compare January 14, 2025 20:29

vladimir-paramuzov reviewed Jan 15, 2025

View reviewed changes

praasz reviewed Jan 15, 2025

View reviewed changes

t-jankowski requested changes Jan 15, 2025

View reviewed changes

.../transformations/include/transformations/common_optimizations/group_normalization_fusion.hpp Outdated Show resolved Hide resolved

jhajducz force-pushed the private/jhajducz/gpu_gn_fusion branch 3 times, most recently from 9cf1017 to 8382572 Compare January 16, 2025 20:23

vladimir-paramuzov reviewed Jan 17, 2025

View reviewed changes

jhajducz force-pushed the private/jhajducz/gpu_gn_fusion branch 2 times, most recently from 215c308 to 75e1dce Compare January 19, 2025 22:32

jhajducz added 24 commits January 20, 2025 00:30

Make conversion of basic data type to element type a constexpr

c163459

Add GroupNormalization fusion to common optimizations

47a7c1b

Add GroupNormalization fusion tests

74c49fc

Enable GroupNormalization fusion pass in GPU plugin

d645f47

Update copyright notice

c440b4e

Revert "Make conversion of basic data type to element type a constexpr"

472af43

This reverts commit 217bcdc.

Refactor GroupNormalizationFusion tests to avoid changes in core API

05ecf45

Remove GPU plugin specific GroupNormComposition pass

c6afa44

Fix RTTI macro in GroupNormalizationFusion header file

028b669

Override TestBody() method in GroupNormalizationFusionTestsFixture

de25f2d

Explain meaning of GroupNormalizationFusion tests parameters

4d069b5

Require providing correct group norm gamma & beta shapes in positive …

3445391

…GroupNormalizationFusion tests

Use dedicated Constant ctor to create scalar constants in GroupNormal…

0bb6137

…izationFusion tests

Avoid Shape->PartialShape conversion for in/out tensors in GroupNorma…

c991ed9

…lizationFusion tests

Use global testing namespace in GroupNormalizationFusion tests

c9631ea

Another update of copyright notice

bb76a51

Use const references where possible in GroupNormalizationFusion pass

45a372e

Move GroupNormalizationFusion after MVNFusion pass in GPU plugin tran…

a113897

…sformations pipeline

Use OV ptr cast for MVN in GroupNormalizationFusion pass

e340a54

Add 5d and 6d cases to GroupNormalizationFusion tests + fix formatting

8a7cddb

Use predicates for type & shape checks that don't depend on other nod…

721c99d

…es in GroupNormalizationFusion pass

Use ov::pass::pattern namespace in GroupNormalizationFusion pass

8ceaa1c

Remove redundant has_integral_type predicate from GroupNormalizationF…

63a3fdd

…usion pass

Simplify accessing nodes partial shapes in GroupNormalizationFusion pass

c163002

jhajducz force-pushed the private/jhajducz/gpu_gn_fusion branch from 75e1dce to c163002 Compare January 19, 2025 23:30

Fix typo in one of types in GroupNormalizationFusion tests

9d9c93d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TRANSFORMATIONS][GPU] Add GroupNormalization fusion to common optimizations #28387

[TRANSFORMATIONS][GPU] Add GroupNormalization fusion to common optimizations #28387

jhajducz commented Jan 11, 2025 •

edited

Loading

mlukasze commented Jan 13, 2025

t-jankowski left a comment

praasz commented Jan 13, 2025

vladimir-paramuzov Jan 17, 2025

jhajducz Jan 17, 2025

vladimir-paramuzov Jan 17, 2025

jhajducz Jan 17, 2025

e-ddykim commented Jan 20, 2025

jhajducz commented Jan 20, 2025

[TRANSFORMATIONS][GPU] Add GroupNormalization fusion to common optimizations #28387

Are you sure you want to change the base?

[TRANSFORMATIONS][GPU] Add GroupNormalization fusion to common optimizations #28387

Conversation

jhajducz commented Jan 11, 2025 • edited Loading

Details:

Tickets:

mlukasze commented Jan 13, 2025

t-jankowski left a comment

Choose a reason for hiding this comment

praasz commented Jan 13, 2025

vladimir-paramuzov Jan 17, 2025

Choose a reason for hiding this comment

jhajducz Jan 17, 2025

Choose a reason for hiding this comment

vladimir-paramuzov Jan 17, 2025

Choose a reason for hiding this comment

jhajducz Jan 17, 2025

Choose a reason for hiding this comment

e-ddykim commented Jan 20, 2025

jhajducz commented Jan 20, 2025

jhajducz commented Jan 11, 2025 •

edited

Loading