[ENH]: Make Collections Casters more customizable.

[Skylion007]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

Problem description

We need a better way to customize the return type of functions. Specifically, we need a way to control the types of casting behavior of nested collections AND have it be function specific. There are use cases where instead of converting to unicode strings, we may want to convert to bytes, or we may want to auto-freeze all sets into frozen sets etc.

For simple use objects, we can just wrap them in the appropiate python types or do the casting in a lambda wrapper, but doing this at scale proves difficult. See the lengthy discussion on 3838. Adding more, return type flags is a non-scalable way of solving this problem.

Currently, all the list, set, etc casters call make_casters on their keys or values to get the casters needed for the function. All we need to do is a templated_way to override that make_caster template with our own templated derivatives. Specifically, the user should be able to specify the way of selecting the caster.

See an example here:

pybind11/include/pybind11/stl.h

Line 143 in c4e2952

using value_conv = make_caster<Value>;

Specifically, this can be fixed if we add more optional template parameters to make_caster in a optional backwards compatible way. Specifically, we need a way to define a custom make_caster recursively for all list, set, and map types. Adding this type of hook would solve most of the issues with casting collections, the user would just call their own custom make_caster which could forward optional args to list, set, map, etc casters.

To make this painless, we would make list_caster, set_caster, map_caster etc more customizable and allow users to specify their own custom make_caster template for the key and value converters.

Thoughts @rwgk @henryiii @wangxf123456 ?

Reproducible example code

No response