Add a viewport UI widget

[chompaa]

Objective

Add a viewport widget.

Solution

• Add a new ViewportNode component to turn a UI node into a viewport.
• Add viewport_picking to pass pointer inputs from other pointers to the viewport's pointer.
  • Notably, this is somewhat functionally different from the viewport widget in the editor prototype, which just moves the pointer's location onto the render target. Viewport widgets have their own pointers.
  • Care is taken to handle dragging in and out of viewports.
• Add update_viewport_render_target_size to update the viewport node's render target's size if the node size changes.
• Feature gate picking-related viewport items behind bevy_ui_picking_backend.

Testing

I've been using an example I made to test the widget (and added it as viewport_node):

Code

//! A simple scene to demonstrate spawning a viewport widget. The example will demonstrate how to
//! pick entities visible in the widget's view.

use bevy::picking::pointer::PointerInteraction;
use bevy::prelude::*;

use bevy::ui::widget::ViewportNode;
use bevy::{
    image::{TextureFormatPixelInfo, Volume},
    window::PrimaryWindow,
};
use bevy_render::{
    camera::RenderTarget,
    render_resource::{
        Extent3d, TextureDescriptor, TextureDimension, TextureFormat, TextureUsages,
    },
};

fn main() {
    App::new()
        .add_plugins((DefaultPlugins, MeshPickingPlugin))
        .add_systems(Startup, test)
        .add_systems(Update, draw_mesh_intersections)
        .run();
}

#[derive(Component, Reflect, Debug)]
#[reflect(Component)]
struct Shape;

fn test(
    mut commands: Commands,
    window: Query<&Window, With<PrimaryWindow>>,
    mut images: ResMut<Assets<Image>>,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<StandardMaterial>>,
) {
    // Spawn a UI camera
    commands.spawn(Camera3d::default());

    // Set up an texture for the 3D camera to render to
    let window = window.get_single().unwrap();
    let window_size = window.physical_size();
    let size = Extent3d {
        width: window_size.x,
        height: window_size.y,
        ..default()
    };
    let format = TextureFormat::Bgra8UnormSrgb;
    let image = Image {
        data: Some(vec![0; size.volume() * format.pixel_size()]),
        texture_descriptor: TextureDescriptor {
            label: None,
            size,
            dimension: TextureDimension::D2,
            format,
            mip_level_count: 1,
            sample_count: 1,
            usage: TextureUsages::TEXTURE_BINDING
                | TextureUsages::COPY_DST
                | TextureUsages::RENDER_ATTACHMENT,
            view_formats: &[],
        },
        ..default()
    };
    let image_handle = images.add(image);

    // Spawn the 3D camera
    let camera = commands
        .spawn((
            Camera3d::default(),
            Camera {
                // Render this camera before our UI camera
                order: -1,
                target: RenderTarget::Image(image_handle.clone().into()),
                ..default()
            },
        ))
        .id();

    // Spawn something for the 3D camera to look at
    commands
        .spawn((
            Mesh3d(meshes.add(Cuboid::new(5.0, 5.0, 5.0))),
            MeshMaterial3d(materials.add(Color::WHITE)),
            Transform::from_xyz(0.0, 0.0, -10.0),
            Shape,
        ))
        // We can observe pointer events on our objects as normal, the
        // `bevy::ui::widgets::viewport_picking` system will take care of ensuring our viewport
        // clicks pass through
        .observe(on_drag_cuboid);

    // Spawn our viewport widget
    commands
        .spawn((
            Node {
                position_type: PositionType::Absolute,
                top: Val::Px(50.0),
                left: Val::Px(50.0),
                width: Val::Px(200.0),
                height: Val::Px(200.0),
                border: UiRect::all(Val::Px(5.0)),
                ..default()
            },
            BorderColor(Color::WHITE),
            ViewportNode::new(camera),
        ))
        .observe(on_drag_viewport);
}

fn on_drag_viewport(drag: Trigger<Pointer<Drag>>, mut node_query: Query<&mut Node>) {
    if matches!(drag.button, PointerButton::Secondary) {
        let mut node = node_query.get_mut(drag.target()).unwrap();

        if let (Val::Px(top), Val::Px(left)) = (node.top, node.left) {
            node.left = Val::Px(left + drag.delta.x);
            node.top = Val::Px(top + drag.delta.y);
        };
    }
}

fn on_drag_cuboid(drag: Trigger<Pointer<Drag>>, mut transform_query: Query<&mut Transform>) {
    if matches!(drag.button, PointerButton::Primary) {
        let mut transform = transform_query.get_mut(drag.target()).unwrap();
        transform.rotate_y(drag.delta.x * 0.02);
        transform.rotate_x(drag.delta.y * 0.02);
    }
}

fn draw_mesh_intersections(
    pointers: Query<&PointerInteraction>,
    untargetable: Query<Entity, Without<Shape>>,
    mut gizmos: Gizmos,
) {
    for (point, normal) in pointers
        .iter()
        .flat_map(|interaction| interaction.iter())
        .filter_map(|(entity, hit)| {
            if !untargetable.contains(*entity) {
                hit.position.zip(hit.normal)
            } else {
                None
            }
        })
    {
        gizmos.arrow(point, point + normal.normalize() * 0.5, Color::WHITE);
    }
}

Showcase

2025-01-11.13-49-15.mp4

Open Questions

• Not sure whether the entire widget should be feature gated behind bevy_ui_picking_backend or not? I chose a partial approach since maybe someone will want to use the widget without any picking being involved.
• Is PickSet::Last the expected set for viewport_picking? Perhaps PickSet::Input is more suited.
• Can dragged_last_frame be removed in favor of a better dragging check? Another option that comes to mind is reading Drag and DragEnd events, but this seems messier.

[Atlas16A]

Code wise seems good to me, just needs updating to deal with conflicts.

[ickshonpe]

The demo works really well, just think the UI side can be simplified a bit.
The image should just be shown on the viewport node, the child node isn't necessary I think. And instead of using an ImageNode, you can add an extraction system to the UI extraction schedule like:

pub fn extract_viewport_nodes_system(
    mut commands: Commands,
    mut extracted_uinodes: ResMut<ExtractedUiNodes>,
    camera_query: Extract<Query<&Camera>>,
    uinode_query: Extract<
        Query<(
            Entity,
            &ComputedNode,
            &GlobalTransform,
            &InheritedVisibility,
            Option<&CalculatedClip>,
            Option<&UiTargetCamera>,
            &ViewportNode,
        )>,
    >,
    camera_map: Extract<UiCameraMap>,
) {
    let mut camera_mapper = camera_map.get_mapper();
    for (entity, uinode, transform, inherited_visibility, clip, camera, viewport_node) in
        &uinode_query
    {
        // Skip invisible images
        if !inherited_visibility.get() || uinode.is_empty() {
            continue;
        }

        let Some(extracted_camera_entity) = camera_mapper.map(camera) else {
            continue;
        };

        let Some(image) = camera_query
            .get(viewport_node.camera)
            .ok()
            .and_then(|camera| camera.target.as_image())
        else {
            continue;
        };

        extracted_uinodes.uinodes.push(ExtractedUiNode {
            render_entity: commands.spawn(TemporaryRenderEntity).id(),
            stack_index: uinode.stack_index,
            color: LinearRgba::WHITE,
            rect: Rect {
                min: Vec2::ZERO,
                max: uinode.size,
            },
            clip: clip.map(|clip| clip.clip),
            image: image.id(),
            extracted_camera_entity,
            item: ExtractedUiItem::Node {
                atlas_scaling: None,
                transform: transform.compute_matrix(),
                flip_x: false,
                flip_y: false,
                border: uinode.border(),
                border_radius: uinode.border_radius(),
                node_type: NodeType::Rect,
            },
            main_entity: entity.into(),
        });
    }
}

Then there's no need for the user to pass in the image handle.

[Atlas16A]

Looks good to my inexperienced eyes

[alice-i-cecile]

I'm happy enough with this to start experimenting with it.

[github-actions]

It looks like your PR has been selected for a highlight in the next release blog post, but you didn't provide a release note.

Please review the instructions for writing release notes, then expand or revise the content in the release notes directory to showcase your changes.

[NthTensor]

Kicked this back to Waiting-On-Author due to lack of release note (sorry!)

Just a basic draft is fine. You may also want to consider linking to the video, so that it's easy to find and embed later (do not commit the video to the repo).

https://private-user-images.githubusercontent.com/26204416/402285264-39f44eac-2c2a-4fd9-a606-04171f806dc1.mp4?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDU4NTY4MDgsIm5iZiI6MTc0NTg1NjUwOCwicGF0aCI6Ii8yNjIwNDQxNi80MDIyODUyNjQtMzlmNDRlYWMtMmMyYS00ZmQ5LWE2MDYtMDQxNzFmODA2ZGMxLm1wND9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA0MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNDI4VDE2MDgyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg0ZDU0OGFmM2Q3NTJmOWJkNDYzODMxNjkyOTBlYzFmNmQ2YWUzMGMzMjJjMjFiZWI0ZmY3ZjZkMjNiMzA5NzkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.DXec6l2SYDIpSCRssEB4o3er7ib3jUQ9t9fvjdY3hYw

[ickshonpe]

Looks good. Don't see any further problems, eager to see this merged.

[chompaa]

Release notes are in! Marking this as Ready-For-Final-Review.

[NthTensor]

Release notes look good.