fix to jamba config, asserting attention and expert offset

[ErezSC42]

What does this PR do?

Ensures instanced models are supported by adding assertions to attention offset and expert offset values

[vasqu]

I think it would be more appropriate to move this to a separate (private) function and instead of an assertion error, raise a ValueError.

[vasqu]

Ah and maybe add a simple test to check if the expected errors are raised 👀

[vasqu]

Just two small nits:

• Could you change t_, it's not really descriptive.
• Maybe move it under JambaConfig.

[vasqu]

Jamba, cc @ArthurZucker

[ErezSC42]

I changed t_ to a be more descriptive, but since the _check_supported_offset function is called inside JambaConfig's init, it cannot be a method defined inside the class

[vasqu]

Thank you!

For _check_supported_offset it should be possible, see for reference:

transformers/src/transformers/models/olmo/configuration_olmo.py

Lines 150 to 181 in f38590d

	self._rope_scaling_validation()
	self.attention_bias = attention_bias
	self.attention_dropout = attention_dropout
	self.clip_qkv = clip_qkv
	super().__init__(
	pad_token_id=pad_token_id,
	bos_token_id=bos_token_id,
	eos_token_id=eos_token_id,
	tie_word_embeddings=tie_word_embeddings,
	**kwargs,
	)
	def _rope_scaling_validation(self):
	"""
	Validate the `rope_scaling` configuration.
	"""
	if self.rope_scaling is None:
	return
	if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 2:
	raise ValueError(
	"`rope_scaling` must be a dictionary with two fields, `type` and `factor`, " f"got {self.rope_scaling}"
	)
	rope_scaling_type = self.rope_scaling.get("type", None)
	rope_scaling_factor = self.rope_scaling.get("factor", None)
	if rope_scaling_type is None or rope_scaling_type not in ["linear", "dynamic"]:
	raise ValueError(
	f"`rope_scaling`'s type field must be one of ['linear', 'dynamic'], got {rope_scaling_type}"
	)
	if rope_scaling_factor is None or not isinstance(rope_scaling_factor, float) or rope_scaling_factor <= 1.0:
	raise ValueError(f"`rope_scaling`'s factor field must be a float > 1, got {rope_scaling_factor}")

At least that's what I had in mind.

[ErezSC42]

my bad, it should be working now

[vasqu]

You can run make style which should fix the remaining issues. Thx for bearing with me :)

[ErezSC42]

Hey, all the tests passed, is this PR good for merging?

[vasqu]

It needs a review from a core maintainer, so we can just wait.

[vasqu]

Maybe cc @amyeroberts

[amyeroberts]

LGTM - thanks for adding!