Optimize Miri for `test_validate_cast_and_convert_metadata`

Currently, [Miri tests are failing](https://github.com/google/zerocopy/actions/runs/6228577863/job/16905623439) (both on main and in PRs) due to `test_validate_cast_and_convert_metadata` taking too long to run. We need to optimize this under Miri to unblock CI.

