[Fix] Fix the print function of AdaptiveTokenMask. (#404)

Seven-Streams · web-flow · commit 5803d9ae9019 · 2025-08-20T22:28:54.000-04:00
This PR fixes the print function of adaptivetokenmask when its type is
`kDynamicBitset`. This is helpful for development.

Signed-off-by: Yuchuan &lt;blemiade_qinchuan@sjtu.edu.cn&gt;

Signed-off-by: Yuchuan &lt;blemiade_qinchuan@sjtu.edu.cn&gt;
diff --git a/cpp/compiled_grammar.cc b/cpp/compiled_grammar.cc
@@ -84,7 +84,7 @@ std::string AdaptiveTokenMask::Print(const TokenizerInfo& tokenizer_info) const
       if (uncertain_indices_set.count(i)) {
         continue;
       }
-      if (accepted_bitset[i]) {
+      if (accepted_bitset[sorted_decoded_vocab[i].first]) {
         accepted_indices.push_back(i);
       } else {
         rejected_indices.push_back(i);

Original file line number	Diff line number	Diff line change
`@@ -84,7 +84,7 @@ std::string AdaptiveTokenMask::Print(const TokenizerInfo& tokenizer_info) const`
`84`	`84`	`if (uncertain_indices_set.count(i)) {`
`85`	`85`	`continue;`
`86`	`86`	`}`
`87`		`- if (accepted_bitset[i]) {`
	`87`	`+ if (accepted_bitset[sorted_decoded_vocab[i].first]) {`
`88`	`88`	`accepted_indices.push_back(i);`
`89`	`89`	`} else {`
`90`	`90`	`rejected_indices.push_back(i);`