Diversity of Realms / Why do we need incumbent settings object for the callbacks

[dSalieri]

I didn't know how to better make my title, ok let it be so it is.

Not so long ago I looked into the whatwg specification in order to find out the specific algorithms of the host procedures referred to by ecma262. And some steps in these algorithms have misled me.

For example, there are such concepts as entry, incumbent, current and relevant. There is an example that demonstrates these terms. Moreover, there is even a description of these concepts, but to be honest they do not explain, but only add even more confusion. It is difficult to understand the difference clearly between incumbent, current and relevant.

In the ecma262 specification, it is clearly stated about the current Realm that it is a Realm component of the running execution context (everything is clear).

• What is the incumbent Realm?
• And what is the relevant Realm?

We need a clear border that can show their difference.

If possible, it would be nice to show the difference of these terms with a different number of Realms to see when the incumbent Realm === current Realm, for example. And when incumbent Realm !== current Realm. According to the whatwg specification, this is difficult to understand.

---

If we consider the algorithms that interested me, I didn't quite understand how the incumbent setting object affects the callback functions that will be called as micro-tasks. There is even a section of Promise on mdn describing this object, but its meaning has not been disclosed, so far, its impact on the document on Realm is unclear. What prevents us from calling this function from then just with an active context object without an incumbent setting object (I'm speaking about field: [[HostDefined]]: { [[IncumbentSettings]] } in the HostMakeJobCallback)

It's about algorithms:

• HostMakeJobCallback (callable)
• HostCallJobCallback (callback, V, argumentsList)

For the first one, gets incumbent setting object and when time comes to return value it places that incumbent object into JobCallback Record.
For the second one, uses incumbent setting object for the preparation of run a callback

So I can't to understand what is incumbent, I don't understand its affection for calling callback functions. Could you clarify this thing?

---

In conclusion, it is not recommended to use the entry and incumbent concept in the specification itself.

The incumbent and entry concepts should not be used by new specifications, as they are excessively complicated and unintuitive to work with. We are working to remove almost all existing uses from the platform: see issue #1430 for incumbent, and issue #1431 for entry.

Then why is this concept used in relation to callback functions from promises and Web IDL?

[dSalieri]

Could you @bzbarsky clarify this?

[bzbarsky]

I really don't know how to clarify this more clearly than the example in the spec, which clearly shows the differences between the concepts... What part of the example was not clear?

In the ecma262 world there is no "relevant Realm": those are attached to what are called "platform objects", which are the objects defined in Web IDL. The "relevant Realm" for each platform object is supposed to be part of the object's definition. In practice, specs are terrible at defining what it should be for a given object, though there are some cases where it's well-defined (e.g. if an object is created via a constructor call, it will be the Realm of the constructor, and this is defined in Web IDL). In particular, a "relevant Realm" is a property of an object, not ambient state. Entry/incumbent/current realms are all ambient state.

For a case when incumbent Realm != current Realm, consider something like this:

   window.frames[0].document.getElementById("foo");

Here the current Realm inside the call to getElementById will be the realm of the subframe, but the incumbent Realm will be the realm of the toplevel window. Basically, it differs from "current Realm" in that calling built-in functions changes the current Realm but does not change the incumbent Realm.

I didn't quite understand how the incumbent setting object affects the callback functions that will be called as micro-tasks

The important part is that in some cases the incumbent realm is used to identify "who called this function". That works fine if the function is called directly, as in my getElementById call above. But if I did:

  window.frames[0].setTimeout(window.frames[0].document.getElementById.bind(window.frames[0].document), 0, "foo");

then what happens? The rules about callbacks in the HTML and Web IDL specs are designed to ensure that the incumbent in this case will be the same as in the simpler example of a direct call above, so you can't spoof the "identity of the caller".

Similar considerations apply when using a function that cares about its incumbent Realm as a Promise callback. So if I do:

Promise.resolve().then(window.frames[0].document.getElementById.bind(window.frames[0].document, "foo"));

that should lead to the same observable behavior as the "simple" call.

[bzbarsky]

As for why incumbent settings object is needed at all, it's used in a few specification algorithms, and hence must be correct when any functions that would invoke those algorithms are called:

• https://html.spec.whatwg.org/#navigate
• https://html.spec.whatwg.org/#window-post-message-steps

[bzbarsky]

One other thing. Above you say:

There is an example that demonstrates these terms.

but I count 5 examples in the spec, one illustrating the difference between all four things, one highlighting what the "relevant settings object" means, and three explaining incumbent settings object, including why it exists and what would go wrong if it did not exist...

[dSalieri]

@bzbarsky Well, I've spent a lot of time for understanding incumbent conception and I got it that I completely don't understand following things:

First:

Here, the topmost script-having execution context is the topmost entry of the JavaScript execution context stack that has a non-null ScriptOrModule component, or null if there is no such entry in the JavaScript execution context stack.

This definition of the topmost script-having execution context makes you confused because you may understand these by 2 ways:

1. Finding last execution context in the stack (last means placing on the top of stack). Execution Context may not has in the field ScriptOrModule as value Module Record or Script Record. In that case we use Module Record or Script Record or null value.
2. Finding execution context that must be on the top of the stack, but if execution context at the top has ScriptOrModule with value null, then look at previous execution context until we will not get execution context with Module Record or Script Record value, otherwise it will be null value.

Which one is correct?

Second:

With all this in place, the incumbent settings object is determined as follows:

1. Let context be the topmost script-having execution context.
2. If context is null, or if context's skip-when-determining-incumbent counter is greater than zero, then:
   1. Assert: the backup incumbent settings object stack is not empty.

   This assert would fail if you try to obtain the incumbent settings object from inside an algorithm that was triggered neither by calling scripts nor by Web IDL invoking a callback. For example, it would trigger if you tried to obtain the incumbent settings object inside an algorithm that ran periodically as part of the event loop, with no involvement of author code. In such cases the incumbent concept cannot be used.

   2. Return the topmost entry of the backup incumbent settings object stack.
3. Return context's Realm component's settings object.

As you can see due with misunderstanding step one, I can't say exactly what will be next. Also I don't even understand comment in substeps; obviously need example of code for clarify.

And also I would like to see cases of code that demonstrate step 2.

---

Real intention of all of that is understand how does work planing tasks by promise conception.

When you are evaluating algorithm of promise's then, this runs HostMakeJobCallback.
Now you should see steps of whatwg specification. I know good ecma mechanisms but what concerns whatwg spec I know it worse (a lot of complicated mechanisms).

Ok, when you're opening HostMakeJobCallback in step one you'll see incumbent settings object and in the second you'll see active script. To be honest, I don't understand what incumbent settings object and active script for. Are we really need in each of them (again need examples of code)? Both will be stored into (active script will be used for creation new execution context that will be stored into variable script execution context)

JobCallback Record {
   [[Callback]]: callable, 
   [[HostDefined]]: {
      [[IncumbentSettings]]: incumbent settings object, 
      [[ActiveScriptContext]]: script execution context
   }
}

Isn't just the [[Callback]] field not enough? It has also [[Realm]] field.

With making HostMakeJobCallback it is over. But here coming second interesting part it is HostCallJobCallback. Just take a look at the first step and the seond step it is [[IncumbentSettings]] and [[ActiveScriptContext]] accordingly. [[IncumbentSettings]] is used in Prepare to run a callback and Clean up after running a callback (don't understand what actualy preparing and cleaning do). [[ActiveScriptContext]] is used as outer execution context. But in the step 5 we see ecma operation Call, I know exactly that this operation will create new execution context for our callback function placed in then method as argument, so what [[ActiveScriptContext]] for?

I did a huge amount of questions (sorry for that). I hope I'll get answers on my questions. Examples of real code would help more.

[bzbarsky]

First:
...
Which one is correct?

The second. I can see how it could It could be confusing, especially to non-native English speakers, because it relies on the English grammatical structure to disambiguate between the two options... Maybe it could be phrased a bit more precisely as:

"Here, the topmost script-having execution context is the topmost of the entries in the JavaScript execution context stack that have a non-null ScriptOrModule component, or null if there are no such entries in the JavaScript execution context stack."

Please file issues to get the wording clarified as needed.

And also I would like to see cases of code that demonstrate step 2.

I gave some examples above, no?

window.frames[0].setTimeout(window.frames[0].document.getElementById.bind(window.frames[0].document), 0, "foo");

Inside the getElementById function if it needed the incumbent settings object, we would be in step 2, because there is no script running at all when that function is called: it's all built-in functions.

I don't understand what incumbent settings object and active script for.

Fundamentally, "incumbent settings object" is "who called this built-in function?"

Have you tried looking at the ways incumbent settings object gets used, then trying to match those uses with the examples in the HTML specification where all the settings objects are different, to see how behavior would change if a different settings object were used?

For the rest... I have not been working on browsers for over 2 years now, and I have paged out the complications about the (very complicated, as you say) Promise bits in the ES spec. I am not going to spend several more hours trying to guess what you mean by "examples of real code", more so than the examples already present in the HTML spec...

[dSalieri]

@bzbarsky finally I've investigated incumbent settings object (without his inside structure). I noticed that it is used in following cases:

• inside postMessage algorithm
• then callbacks that uses host hooks
• navigation case

As I can see incumbent settings object is used in the postMessage for source when we will catch event. Also incumbent settings object is used into host-hooks related with then callbacks (unclear purpose incumbents here). And incumbents settings object multiply appears in navigation (I think it's main purpose of incumbents).

There is incumbent settings object tracking on the mdn. Explanation with examples doesn't explain what is incumbent settings object tracking. More than that last example says that code in message event will be performed if browser has incumbent settings object tracking. But the main problem is callback function that not using incumbent (I suspect that this done for the code inside callback tied with postMessage and navigation case; isn't it?). In the end how should I see that tracking in the mdn article about incumbent settings object tracking. What does incumbent settings object actually do in the promises?

And one thing, there is following quote

This is perhaps the only justifiable use of the incumbent concept on the web platform; in all other cases the consequences of using it are simply confusing and we hope to one day switch them to use current or relevant as appropriate.

What will be example of the real code (I mean practical example)?

[bzbarsky]

The point of incumbent object tracking is that if some code queues a function for later execution (e.g. via Promise then, setTimeout, or any other mechanism) when that function later runs the incumbent needs to be exactly the same as it would have been if the code had just called the function directly instead of queueing it up.

So anything that does async execution needs to track what the incumbent was when the async operation was queued up and make sure that's the incumbent when the function is actually called.

[dSalieri]

@bzbarsky this is neat and good explanation of the incumbent conception. I like it.

Have you seen mdn article with incumbent section? How do these examples demonstarte incumbent setings object? I am definitely don't understand them, because they don't reflect how incumbent settings object changes when we will change supporting browser (firefox) to not supporting browser (opera). What could you say about this?

And for the rest: caniuse incumbent - how is it expressed? Example?

[bzbarsky]

Have you seen mdn article with incumbent section?

@dSalieri I glanced at it, but I did not have time to read it carefully. See above about my involvement with browsers.

I have no idea what the caniuse page is about, but fundamentally you would need to pick a behavior that depends on incumbent (postMessage or navigation), trigger it off a Promise callback (as in, pass a bound native function to then) and see how it behaves.

[dSalieri]

@bzbarsky I remember that you are not working with browsers already how 2 years later. But I don't know who can give answers to me. Only you doing it.

Well, I was making as you wrote, but I don't see any difference. Here is sandbox where I created example. I am confused. Could you demonstrate this interesting behaviour that will be look different among firefox and rest browsers?

[bzbarsky]

In your example, Firefox says "This text from: a.html" and so does Chrome, while Safari says "This text from: b.html".

[dSalieri]

@bzbarsky I haven't Safari, but have Chrome, Firefox, Edge, Maxthon and they show all the same. I even tested in 63 version of Chrome and again result the same. But I've checked Safari on the browserstack version 14.1 and I received your behaviour. Strange thing. Seems like a bug in Safari, IMO.

[bzbarsky]

Chrome, Edge and Maxthon are all the same browser for purposes of this discussion.

Anyway, there were some cases where Chrome and Firefox differed in terms of incumbent handling. @domenic might recall what they were, but at this point I really can't recall what they were. As I said, all this has mostly been paged out. And it's possible that those divergences have been eliminated in the last two years.

And yes, per spec as currently intended the Safari behavior is a bug.

[dSalieri]

@bzbarsky you know this all is very much wonder me. So as we can see almost all browsers support incumbent settings object tracking, right? Then this picture opposite the title Job-related host hooks in whatwg spec is incorrect in relative to browsers.

Who is right browsers or specification?

[bzbarsky]

This is not "specification". It's a pointer to a wiki page, which could be out of date.

I suggest asking the Chrome team directly what they do or don't support here instead of trying to guess.

[dSalieri]

This is not "specification". It's a pointer to a wiki page, which could be out of date.

Didn't know, I thought it is official declaration of whatwg spec.

I suggest asking the Chrome team directly what they do or don't support here instead of trying to guess.

@bzbarsky how can I do this, I mean where?

[bzbarsky]

If this is part of ongoing standards work, by talking to the Chrome representatives in the relevant standards group.