Lookups broken for `typos` with 2.8.1

[epage]

Note: I'm still working on a root cause (whether with my code, unicase, or phf) but wanted to communicate this early

I've narrowed down the failure in crate-ci/typos#1202 to the unicase 2.8.0 to 2.8.1 upgrade.

  ---- dict::test::test_dict_to_varcon stdout ----
  thread 'dict::test::test_dict_to_varcon' panicked at crates/typos-cli/src/dict.rs:379:9:
  assertion `left == right` failed
    left: None
   right: Some(Corrections(["finalises"]))

The test is

    #[cfg(feature = "dict")]
    #[test]
    fn test_dict_correct() {
        let dict = BuiltIn::new(crate::config::Locale::default());
        let correction = dict.correct_word(typos::tokens::Word::new_unchecked(
            "finallizes",
            Case::Lower,
            0,
        ));
        assert_eq!(
            correction,
            Some(Status::Corrections(vec!["finalizes".into()]))
        );
    }

This is looking up finallizes in a PHF Map. I use wrapper types (InsensitiveStr and InsensitiveAscii) to avoid calling a const fn on thousands of items which can dramatically slow down compilation times. These delegate to unicase::UniCase and unicase::Ascii which I believe are supposed to hash to the same thing for the same input.

[epage]

I've tried to create a minimal reproduction case but have failed so far

#!/usr/bin/env nargo
---
package.edition = "2021"
[dependencies]
unicase = "=2.8.1"
dictgen = { version = "*", features = ["map"] }
phf = "*"
---

fn main() {
    let word = "finallizes";

    let aword = unicase::Ascii::new(word);
    dbg!(calculate_hash(&aword));
    let uword = unicase::UniCase::new(word);
    dbg!(calculate_hash(&uword));
    dbg!(calculate_phf_hash(&uword));
    dbg!(aword == uword);

    let aword = dictgen::InsensitiveAscii(word);
    dbg!(calculate_hash(&aword));
    dbg!(calculate_phf_hash(&aword));
    let uword = dictgen::InsensitiveStr::Ascii(word);
    dbg!(calculate_hash(&uword));
    dbg!(calculate_phf_hash(&uword));
}

fn calculate_hash<T: std::hash::Hash>(t: &T) -> u64 {
    use std::hash::Hasher as _;

    let mut s = std::hash::DefaultHasher::new();
    t.hash(&mut s);
    s.finish()
}

fn calculate_phf_hash<T: phf::PhfHash>(t: &T) -> u64 {
    use std::hash::Hasher as _;
    use phf::PhfHash as _;

    let mut s = std::hash::DefaultHasher::new();
    t.phf_hash(&mut s);
    s.finish()
}

❯ ./typos-1202.rs
   Compiling typos-1202 v0.0.0 (/home/epage/src/personal/dump)
warning: unused import: `phf::PhfHash`
  --> /home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:39:9
   |
39 |     use phf::PhfHash as _;
   |         ^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: `typos-1202` (bin "typos-1202") generated 1 warning
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
     Running `/home/epage/.cargo/target/16/84f26244fb8f49/debug/typos-1202`
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:15:5] calculate_hash(&aword) = 87680493026773805
79
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:17:5] calculate_hash(&uword) = 87680493026773805
79
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:18:5] calculate_phf_hash(&uword) = 8768049302677
380579
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:19:5] aword == uword = true
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:22:5] calculate_hash(&aword) = 87680493026773805
79
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:23:5] calculate_phf_hash(&aword) = 8768049302677
380579
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:25:5] calculate_hash(&uword) = 87680493026773805
79
[/home/epage/.cargo/target/16/84f26244fb8f49/typos-1202.rs:26:5] calculate_phf_hash(&uword) = 8768049302677
380579

[seanmonstar]

The hashing implementation did change from 2.8.0 to 2.8.1, to make sure the operations were consistent if ascii or unicode (they used to differ slightly). Do you need to regenerate the hashes? I see there's a file checked in with a comment at the top "generated by codegen".

[epage]

Thats it; thanks for the tip! I don't know why I overlooked that.