Why are \n generated in the output

[Cgrandjean]

Hello guys,
I would like to train the model with format enforced output. I plan to mask the forced tokens to avoid learning this part.
But i see a lot of \n are added ,these constitute tokens too and i was wondering whats the logic behind that .Why are they added. Are they always there and thats normal,in which case i should mask them or is it something else?

Thanks for all answers!

[aw632]

They're whitespace tokens (\n, \t, \d, etc) which are typically allowed. You can disable them using the CharacterLeveParserConfig.