useFocusOutside fails with iframes

[afercia]

Description

useFocusOutside is supposed to trigger a callback when clicking outside of the node it is bound to. An example of this is the modal dialog component: clicking outside the modal dialog closes it.

This is the most common behavior for modal dialog and I'd say it's what users expect.

However, when the modal dialog contains an iframe, useFocusOutside can't work simply because the events it relies to aren;t triggered when clicking inside the iframe. When doing so, the modal dialog closes unexpectedly.

RIght now, there's one case where a modal dialog contains an iframe: the Classic block modal dialog contains the TInyMCE editor, where the editable area is within an iframe.

To 'solve' the problem, the close on click outside behavior of the Classic block modal dialog has been disabled by not passing the onRequestClose prop (which expects a callback to handle the close behavior).

I'm not sure simply disabling the expected behavior is the best option:

• Users expect the modal to close when clicking outside of it, btw nothing happens.
• The unexpected behavior occurs with any iframe contained within a modal, not just with the Classic block, e..g iframes provided as content by plugins.
• Not setting onRequestClose also disables the close on Escape key behavior.

Rather, it would be best to find a way to make useFocusOutside always work as expected, even when iframes are part of the content of the node it is bound to.

Step-by-step reproduction instructions

• Use latest Gutenber gplugin.
• Edit a post and add a Classic block.
• A modal dialog opens.
• Click outside the modal: observe the modal stays open.
• Press the Escape key: observe the modal stays open.
• Alter the Classic block modal dialog code by adding something like this to the <Modal> props:

onRequestClose={ () =>
	content ? setOpen( false ) : onReplace( [] )
}

• Build and refresh the page.
• Repeat the steps above.
• Click outside the modal: observe the modal now closes.
• Press the Escape key: observe the modal now closes.
• Click within the TInyMCE editable area: observe the modal unexpectedly closes.
• Tab into the TInyMCE editable area: observe the modal unexpectedly closes.

Screenshots, screen recording, code snippet

No response

Environment info

No response

Please confirm that you have searched existing issues in the repo.

Yes

Please confirm that you have tested with all plugins deactivated except Gutenberg.

Yes

[t-hamano]

This issue would be related to #40912.