Tracking issue for custom hashers in HashMap

[alexcrichton]

This is a tracking issue for the unstable hashmap_hasher feature in the standard library. This provides the ability to create a HashMap with a custom hashing implementation that abides by the HashState trait. This has already been used quite a bit in the compiler itself as well as in Servo I believe.

Some notable points to consider:

• Is an extra HashState trait really necessary?
• Is the naming of HashState correct?
• Is the aggressive use of Default appropriate here?
• Can the new constructor be leveraged to create hash maps that use a hasher implementing Default? Right now the new constructor only works with RandomState.
• How ergonomic is it to create a hash map using a custom Hasher implementation? In theory it should be quite ergonomic.

cc @gankro

[tomaka]

I don't know much about hashing functions, but I just want to say that this one is very important for performances.

The SipHasher is very slow, and always eats up between 5% and 20% of the CPU time of each of my projects. By comparison the FnvHasher is around 6 times faster on my machine according to some quick benchmarking.

The purpose of the SipHasher is to protect against DDoSes, but the vast majority of the applications written in Rust never even open a socket.

[bluss]

SipHash looks good on long enough data, but it is slow on small values like integers. Its fixed four round finalization at the end is an example of overhead that is comparatively larger when hashing small values.

[Gankra]

See: https://github.com/shepmaster/twox-hash/blob/master/README.md for scaling comparison.

Based on these results (which I haven't vetted for quality), one can conclude that SipHash is actually a pretty good "general purpose" hasher. Fnv doesn't scale as well to longer strings (> 32 bytes), and XXHash doesn't scale well to smaller strings (< 32 bytes).

[Gankra]

CC @frankmcsherry @pczarn @cgaebel @shepmaster

[bluss]

(XXhash was constructed to be a fast checksum for gigabytes of lz4 data, so that's what it is good at)

[alexcrichton]

If HashState is stabilized then we should also ensure that SipHasher isn't exposed as the default hash algorithm on hash maps. Right now the associated type Hasher in RandomState's HashState implementation exposes this and we'll want to wrap that up in something like DefaultHasher instead.

[Gankra]

Just noting some discussion that was tangentially related to this:

The current hasher infra heavily penalizes Farmhash (and cityhash, and murmurhash), which wants to branch on the size of the input to choose the "optimal" algorithm. Community's current solution is to just buffer in a Vec until finish is called. This appears to be overkill, at least in the case of Farmhash.

However more generally it would be nice (even for SipHash) to be able to to indicate to the hasher "there is only one thing to hash, here it is, now give me the final hash". Sketch of design:

add:

pub trait Hasher {
    ...
    /// Hash only the given bytes, and immediately finalize.
    /// Results are unspecified if other bytes were previously written to this hasher
    fn write_only(&mut self, bytes: &[u8]) -> u64 {
        self.write(bytes);
        self.finish()
    }
}

pub trait Hash {
    ...
    /// Hashes only this value
    fn hash_one_shot<H: Hasher>(&self, state: &mut H) -> u64 {
        self.hash(state);
        state.finish()
    }
}

So far, so pointless. But back-compat!

But now hashers can override write_only to something optimized, and Hash impls can specialize in the following way:

// #[derive]d impl
impl Hash {
   fn hash() { ... } // same-old

   #[if(only_has_one_field)]  // magic
   fn hash_one_shot<H: Hasher>(&self, state: &mut H) -> u64 {
      self.only_field.hash_one_shot(state)
   }
}

And key types like slice, str, primitives, etc can manually impl hash_one_shot:

fn hash_one_shot<H: Hasher>(&self, state: &mut H) -> u64 {
    state.write_only(self.as_slice())
}

However we must be wary of #5257!

In particular, slices and strs must mix in some "bonus" value so that ("aa", "a") does not hash the same as ("a", "aa"). My proposed solution to this is simple, but sketchy: have has_one_shot and hash do different things. one_shot knows it's the only thing in existence, and so doesn't need to mix in the bonus data to prevent concatenation collisions. hash doesn't, so it has to conservatively mix in the bonus data.

This seems really sketchy, but I can't think of a situation where this would actually break things. You will get potentially curious behaviour, where &["hello"] doesn't hash to the same thing as "hello" or ("hello",), but this doesn't seem like a serious problem. They're fundamentally different types! Anyone relying on them hashing the same is in a pretty dubious position. Alternatively, in the case of slices, you could actually have them branch at runtime on their len to determine whether to call hash or hash_one_shot on their elements.

I've been flying around the country and haven't slept though.

[Gankra]

Incidentally, this proposal soft-fixes #27108 (&[u8] and &str would indeed hash_one_shot to equal values).

[ranma42]

To me, the naming of Hasher and HashState is quite unintuitive. I expected the hash state (in many other libs, see openssl and crypto++, named "context") to be updated and finalised (using .write() and .finish()), but they actually belong to the Hasher trait.

[bluss]

@gankro I'd like to explore a simpler model, where the Hash trait remains the same (for backwards compatibility and sanity), and the Hasher is extended. Changing str and slice hashes by default too. A Wip is up at #28044

[ranma42]

@gankro I'm exploring a slightly different design here: https://github.com/ranma42/rust-hash/tree/master/src

Besides from the naming, which can obviously be made more consistent with that currently used in Rust, the main difference is that the stream-oriented part of the code is aggressively inlined. Ideally I would want the implementation of one-shot digest to be automatically generated by the compiler, which should be able to get rid of the streaming overhead simply with constant folding and dead code elimination.

This actually happens in the branch I posted, namely the digest method for [u8] does not contain the tail and ntail variables nor the buffered tail flushing loop. Even better, for u8 the length variable gets optimised away, too.

The usual structure of hash functions makes me wonder whether there is much to gain by having write_u8, write_u16, etc. My main objection is that most hash functions do not consume a single byte, but a block of bytes every "step". This means that some kind of buffering and re-alignment is going to be needed in most cases, probably removing any opportunity for effective optimisations.
I believe that a much more effective approach is to have digest_u8, digest_u16 and so on (in the equivalent of my HashFunction trait, which I believe currently is HashState), because this would ensure that the initialisation and finalisation of the hash context can be folded into the computation and that no re-alignment is needed. Additionally for the primitive types {u,i}_{size,8,16,32,64} the fixed length means that the code for most hash functions would be very easy to automatically optimise by the compiler.
I wonder if it might even make sense to add a dispatch for [u8] slices that optimises a small number of fixed sizes (basically arrays up to a certain size)... I will know after some more experimentation ;)