Skip to content

Comments

Update GroupSizeFallbackMode logic#3705

Merged
ljaljushkin merged 7 commits intoopenvinotoolkit:developfrom
nikita-savelyevv:ns/update-group-size-fallback-logic
Nov 4, 2025
Merged

Update GroupSizeFallbackMode logic#3705
ljaljushkin merged 7 commits intoopenvinotoolkit:developfrom
nikita-savelyevv:ns/update-group-size-fallback-logic

Conversation

@nikita-savelyevv
Copy link
Collaborator

@nikita-savelyevv nikita-savelyevv commented Oct 29, 2025

Changes

  • Use ERROR mode by default, in the error message IGNORE/ADJUST modes are suggested to be used instead.
  • Introduce group size value information to the bit-width distribution table.
  • Set min_adjusted_group_size=32 by default instead of 16, because group size 16 usually results in lower performance compared to int8_asym.

Example compression log when group size is adjusted:

# nncf.compress_weights(
#     model,
#     mode=nncf.CompressWeightsMode.INT4_ASYM,
#     ratio=0.9,
#     advanced_parameters=nncf.AdvancedCompressionParameters(
#         group_size_fallback_mode=nncf.GroupSizeFallbackMode.ADJUST,
#     )
# )

INFO:nncf:Some nodes can't be quantized with the specified group size of 128. Adjusted group size values will be used.
Mixed-Precision assignment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━100% 384/384 • 0:00:09 • 0:00:00
INFO:nncf:Statistics of the bitwidth distribution:
┍━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┑
│ Weight compression mode   │ % all parameters (layers)   │ % ratio-defining parameters (layers)   │
┝━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┥
│ int8_asym, per-channel    │ 23% (224 / 385)             │ 10% (223 / 384)                        │
├───────────────────────────┼─────────────────────────────┼────────────────────────────────────────┤
│ int4_asym, group size 64  │ 2% (30 / 385)               │ 2% (30 / 384)                          │
├───────────────────────────┼─────────────────────────────┼────────────────────────────────────────┤
│ int4_asym, group size 128 │ 75% (131 / 385)             │ 88% (131 / 384)                        │
┕━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┙
Applying Weight Compression ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━100% • 0:00:27 • 0:00:00

Reason for changes

User feedback.

Related tickets

175084

Tests

Updated existing tests.

@nikita-savelyevv nikita-savelyevv requested a review from a team as a code owner October 29, 2025 10:06
@github-actions github-actions bot added the API Public API-impacting changes label Oct 29, 2025
Nikita Savelyev added 2 commits October 29, 2025 11:38
Copy link
Contributor

@ljaljushkin ljaljushkin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @nikita-savelyevv.
This PR should improve UX and stats look more clear now!

One minor suggestion to add , and - in the table:
int8_asym, per-channel
int4_asym, group size 32

@nikita-savelyevv
Copy link
Collaborator Author

Thank you, @nikita-savelyevv. This PR should improve UX and stats look more clear now!

One minor suggestion to add , and - in the table: int8_asym, per-channel int4_asym, group size 32

Thanks for the suggestion! Updated.

@ljaljushkin ljaljushkin merged commit dcbe443 into openvinotoolkit:develop Nov 4, 2025
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

API Public API-impacting changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants