Missing text/selection manipulation primitives

[andreastt]

https://www.w3.org/Bugs/Public/show_bug.cgi?id=29135

w3c.org@masklinn.net:

As far as I could see, the WebDriver spec currently provides very little in terms of emulating textual manipulations.

NOTE: I will use the term "insertion point" to refer to the textual cursor within e.g. a text box, to differentiate it from the "pointer" cursor

Current Provisions

• the entire textual content of an element can be retrieved
• it is possible to [clear] an element or [sendKeys] to it (emulating keyboard input)
• implicitly, the insertion point and selection can be manipulated using actions (click and pointerDown/pointerMove/pointerUp).

Primary Issues

Pointer actions work in term of offsets, but as far as I could tell

• the specification provides no way to perform textual matching and transform that into bounding boxes, thus no way to easily position the insertion point or draw selections
• the specification provides no way to query the insertion point or selection for position or bounding boxe, thus no way to get simple feedback while probing blindly

Use case

Test/demonstrate RTEs or other contenteditable elements, allow cross-platform text insertion within existing textual nodes rather than just around them

Possible solutions?

Rect textRect(needle[, element][, skip])

• would return the same thing as Element Rect ({x, y, width, height} relative to the document element).
  • would only match visible text (so text contained in a visible element)
  • would generate an error if no matching visible text is found?
• needle would be the text to look for, possibly a regex? The specification does not currently use regex anywhere so that might be a bit much.
• skip would probably be necessary as the reference text could occur multiple times in the source.
• a WebElement "root reference" would probably allow easier precise matching and less skipping.
• Testing Chrome, Firefox and Safari on OSX, selecting a glyph requires going through the majority of the glyph so selecting from a textual boundary won't risk selecting the preceding glyph.
• It's somewhat inconvenient for single-letter boundary selections though as there might be need for lots of skipping.
• It doesn't try to count characters/glyphs and thus might help avoid possible confusion issues with respect to code units, normalisation (maybe?), codepoints and glyphs at the interface-level (these concerns may have to be handled at the spec level though).

Unknowns for this possible solution

• would/should it be possible to match text across multiple elements? This is possible for users e.g. my browser's in-page search will find a match for "requests | preferences" on the current page even tough that spans two links and a span in two separate list elements.
• would/should the rect be augmented with the text's container element(s) in the style of a DOM Range? It doesn't seem to make much sense from a user-interaction perspective.

Unsolved

Should it be possible to query the current selection's span/rect as well, independently from arbitrary text? I don't have a use-case for that right now but a "living" user would see the current text selection displayed in the UI so it could make sense.

[andreastt]

Andreas Tolfsen:

This seems somewhat related to bug 29247.

[AlexandreBonneau]

I'm not sure if this is entirely related or if it's the right place to report that, but one use case where the lack of text selection/caret positioning lacks is a very simple one :

1. Set the caret position (or the selection) in the input wherever I want,
2. Input some characters at that position,
   ...and that's it.

Currently I found a way to set the caret position, but this feels like a common use case and perhaps a built-in function (ie. setSelection(element, start, end = null)) would help.

On the other hand, sendKeys just ignores the caret position when it sends keys, so the text selection point is moot until that bug is fixed (because it looks like a bug to me, am I wrong?).

cf. real world use case here and there.

[illicitonion]

You can sendKeys the HOME, END, ARROW_LEFT and ARROW_RIGHT keys, which allows you to set the carat position.

[AlexandreBonneau]

Well, as a workaround to sendKeys resetting the caret position, I tried a few variations of this idea in the original codepen mentioned in the second link I pasted, line 129-131 :

browser.keys(selectors.input0, [browser.keys.LEFT_ARROW, browser.keys.LEFT_ARROW, browser.keys.LEFT_ARROW, browser.keys.LEFT_ARROW, '123']); // Not working

browser.keys(selectors.input0, ['ArrowLeft', 'ArrowLeft', 'ArrowLeft', 'ArrowLeft', '123']); // Not working

browser.keys(selectors.input0, ['\uE012', '\uE012', '\uE012', '\uE012', '123']); // Not working

but somehow none of that worked too.

Either way, I do think sendKeys should not change the caret position, don't you?

[illicitonion]

I could certainly argue that if the element currently has focus, the caret position should not be changed. I can't remember exactly why that isn't the case, but I'm afraid it's unlikely to be changed in the spec at this point... I wonder if anyone else can remember the details here?

[AlexandreBonneau]

@illicitonion In your comment few weeks ago, you were saying :

You can sendKeys the HOME, END, ARROW_LEFT and ARROW_RIGHT keys, which allows you to set the carat position.

Would you by any chance have a working example where a script sends at least on key to an input?

[shs96c]

The selenium "send keys" command sets the caret position to the end of the string to facilitate users doing interesting things in a loop from the local end: early bug reports and usage suggested they'd always want to append content rather than replace what's existing. There's a strong case for what @illicitonion suggested if the focus is already in the element, but it depends on whether we have enough time to get the implementation in place or not.

If you're using the selenium python bindings and marionette, the following should work:

import selenium.webdriver
from selenium.webdriver.common.keys import Keys

d = webdriver.Firefox()
d.get('http://www.google.com')
q = d.find_element(by = 'name', value = 'q')
q.send_keys('Hello world' + Keys.LEFT + Keys.LEFT + Keys.LEFT + Keys.LEFT + Keys.BACKSPACE + 'W')

You should see the search box contain "hello World".

These APIs are a good candidate for Level 2.