Mesh picking offset when the camera's viewport physical_position is non-zero

[Henauxg]

Bevy version

0.16 (commit 199c978)

What you did

Raised by Krzysiu on Discord

Modify the simple_picking example to add a Viewport with an offset:

• Add the following import use bevy_render::camera::Viewport;
• Add the window_query: Query<&Window> parameter to the setup_scene system
• Replace

    commands.spawn((
        Camera3d::default(),
        Transform::from_xyz(-2.5, 4.5, 9.0).looking_at(Vec3::ZERO, Vec3::Y),
    ));

with

    let window = window_query.single().unwrap();
    commands.spawn((
        Camera {
            viewport: Some(Viewport {
                physical_position: UVec2::new(250, 0),
                physical_size: window.physical_size(),
                ..default()
            }),
            ..default()
        },
        Camera3d::default(),
        Transform::from_xyz(-2.5, 4.5, 9.0).looking_at(Vec3::ZERO, Vec3::Y),
    ));

What went wrong

• Picking on the text works as expected to spawn cubes
• Picking the cubes to rotate them is broken. The user needs to pick 250 px to the right of the cubes.

Additional information

• Setting the offset value to something greater like 450, causing the displaced picking zone to land outside of the initial window, means that picking won't work at all on the cubes, even if the window is resized horizontally. (Edit: I guess that this is expected and due to the viewport being clamped to the window size. Shouldn't it try to expand to its original physical_size though ?)
• Krzysiu created another repro (original discovery) with a setup a bit more involved https://github.com/KrzysztofZywiecki/WFC

  It creates two cameras with different viewports and a rectangle that's visible on both. Left camera picks the square correctly, right one does not, and in the console I'm printing rays dictionary that shows weird ray origin on right camera
  

• Vaguely related issue, but for sprites: Trigger<Pointer<_>> events are triggered for different position when using 2 cameras with RenderLayers #18005, with an ongoing sprite related fix PR Use RayMap and RenderLayers in bevy_sprite/picking_backend #18069

[KrzysztofZywiecki]

I discovered that this bug appears in commit 683b08f associeted with PR #17633 . This issue doesn't happen in b574599 that's right before. The only difference was that I had to add a small margin to viewport's physical_size: physical_size: window.physical_size() - UVec2::new(350, 0) because the app panicked otherwise.

My guess is that after this PR, make_ray method in bevy_picking/src/backend.rs corrects cursor physical position to logical position twice, once in the function and the second time in camera.viewport_to_world. This version of make_ray works correctly both with modified example, and my application.

    fn make_ray(
        primary_window_entity: &Query<Entity, With<PrimaryWindow>>,
        camera: &Camera,
        camera_tfm: &GlobalTransform,
        pointer_loc: &PointerLocation,
    ) -> Option<Ray3d> {
        let pointer_loc = pointer_loc.location()?;
        if !pointer_loc.is_in_viewport(camera, primary_window_entity) {
            return None;
        }
        let mut viewport_pos = pointer_loc.position;
        // if let Some(viewport) = &camera.viewport {
        //     let viewport_logical = camera.to_logical(viewport.physical_position)?;
        //     viewport_pos -= viewport_logical;
        // }
        camera.viewport_to_world(camera_tfm, viewport_pos).ok()
    }

[KrzysztofZywiecki]

I did some more testing. Applying change described above seems to fix picking in examples: simple_picking, mesh_picking, I added a simple observer in split_screen to ground plane and tested individual cameras if they can pick and with this change they can - on master only the top left camera can pick, also UI interferes with picking, however sprite_picking with camera viewport offset stays broken with this change.

[Henauxg]

Good find ! Do you want me to whip up a PR for your fix ?
The reason sprites don't work yet is probably due to this: #18069

[KrzysztofZywiecki]

Thanks for clarifying about why sprites don't work. I'll prepare a PR after work to practice doing it, but thank you so much for your initiative 😃