util: Handle null prototype on inspect #22331

antsmartian · 2018-08-15T05:17:14Z

The implementation and texts are taken from the discussion over here:
#22141 (comment), #22141 (comment)

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

antsmartian · 2018-08-15T05:19:43Z

test/parallel/test-util-inspect.js

@@ -1487,7 +1487,6 @@ assert.strictEqual(util.inspect('"\'${a}'), "'\"\\'${a}'");
  [() => {}, '[Function]'],
  [[1, 2], '[ 1, 2 ]'],
  [[, , 5, , , , ], '[ <2 empty items>, 5, <3 empty items> ]'],
-  [{ a: 5 }, '{ a: 5 }'],


Had to remove this test case as this we are checking the message for both cases (with prototype and without prototype). So it was failing with this change. May be I need to do the same test case outside this loop?

None of these tests should work as before if we detect a null prototype on each of these data types. So the test has to be rewritten anyway.

BridgeAR

This is a good start but it should not only check for objects but for any type that we explicitly test for and mark it as having a null prototype.

BridgeAR · 2018-08-15T14:15:13Z

test/parallel/test-util-inspect.js

@@ -1487,7 +1487,6 @@ assert.strictEqual(util.inspect('"\'${a}'), "'\"\\'${a}'");
  [() => {}, '[Function]'],
  [[1, 2], '[ 1, 2 ]'],
  [[, , 5, , , , ], '[ <2 empty items>, 5, <3 empty items> ]'],
-  [{ a: 5 }, '{ a: 5 }'],


None of these tests should work as before if we detect a null prototype on each of these data types. So the test has to be rewritten anyway.

TimothyGu · 2018-08-16T01:26:54Z

I think this solves the superficial case of #22141 where a prototype is null, but not the general problem of deepEqual that we are not displaying the actual reason why the assertion fails. I am particularly not too great of a fan of special casing null.

My 2¢.

antsmartian · 2018-08-16T02:07:29Z

@TimothyGu Yes, but when using strictEqual (or any other assert) we get the appropriate error message when the assertion fails, for example with this patch we get:

AssertionError [ERR_ASSERTION]: Expected inputs to be strictly equal:
+ actual - expected

+ [Object: null prototype] {}
- {}

I'm not sure, what do you mean by:

but not the general problem of deepEqual that we are not displaying the actual reason why the assertion fails

Am I missing something here?

devsnek · 2018-08-16T02:18:32Z

@antsman whatever changes you make here should be made to assertion, not inspection, is Timothy's point. I feel the same way.

antsmartian · 2018-08-16T02:21:52Z

@devsnek Ok got it. @BridgeAR thoughts?

antsmartian · 2018-08-16T02:34:10Z

I guess the real problem here is that, when assert fails with null prototype we are not displaying the right error message (no indication that the object had null prototype). Looking at the source code, I see when there is a assert failure, we do call inspect to build up the diff for the error message. If i'm understanding it correctly, then I feel inspect is the right place to handle this, because it doesn't seems to be an assert issue.

May be I'm wrong here too. Open for thoughts.

BridgeAR · 2018-08-17T15:15:43Z

@TimothyGu @devsnek for me this is not about assert but about the general handling of null prototypes. I would like to know if the object of kind xyz has a prototype or not. Currently util.inspect does not distinguish these cases but IMO that should be the case.

> {}
{}
> Object.create(null)
{} // I would like to know that this is a different object than the one above.

TimothyGu · 2018-08-23T16:25:33Z

I would still say that a solution that works for all prototypes rather than special casing null – even in util.inspect() – is the way to go.

BridgeAR · 2018-08-23T16:31:33Z

@TimothyGu that is the plan. Currently all prototypes are handled in inspect besides the null prototype.

TimothyGu · 2018-08-23T16:34:16Z

For reference, Chrome DevTools use a pseudo-__proto__ property for such purposes.

antsmartian · 2018-08-23T16:54:13Z

@BridgeAR @TimothyGu Just an update. I'm working on the same for handling all prototypes. Will update the PR in a day or two.. :)

antsmartian · 2018-08-25T11:37:25Z

@BridgeAR I have implemented the check on all the types. Can you please have a look at it?

BridgeAR

This is already looking really good! Thanks a lot for looking into it!

Just a few comments and we should make sure the null prototype is also printed in case a tag exists but no constructor.

BridgeAR · 2018-08-25T11:52:33Z

test/parallel/test-util-inspect.js

-  [new SharedArrayBuffer(2), 'SharedArrayBuffer { byteLength: undefined }']
+   '[DataView: null prototype] {\n  byteLength: undefined,\n  ' +
+     'byteOffset: undefined,\n  buffer: undefined }'],
+  [new SharedArrayBuffer(2), '[SharedArrayBuffer : null prototype] ' +


Looks like a typo after SharedArrayBuffer?

Addressed all other comments, but I couldn't able to see any typo 🤔 Sorry could you please explain me, whats wrong here? cc @BridgeAR

See my comment above :-)

BridgeAR · 2018-08-25T11:59:56Z

test/parallel/test-util-inspect.js

+  assert.strictEqual(
+    util.inspect(Object.setPrototypeOf(value, null)),
+    expected
+  );


Please add https://github.com/nodejs/node/blob/cd98c735e879e1218b1c8514ab4218efa841bba4/test/parallel/test-util-inspect.js#L1616-L1620 as well. Otherwise important things would not be tested anymore.

BridgeAR · 2018-08-25T12:03:11Z

test/parallel/test-util-inspect.js

  value.foo = 'bar';
  assert.notStrictEqual(util.inspect(value), expected);
  delete value.foo;
  value[Symbol('foo')] = 'yeah';
  assert.notStrictEqual(util.inspect(value), expected);
 });

+[
+  [new Set([1, 2]), '[Set: null prototype] { 1, 2 }'],


Would you please also add Arrays, Dataviews and ArrayBuffers? Errors, Dates and regular expressions should be handled in a different PR as they do not show subclassing at all and that should be fixed as well.

BridgeAR · 2018-08-25T12:08:00Z

lib/util.js

+function checkNullPrototype(constructor, type) {
+  if (constructor === '')
+    return `[${type}: null prototype]`;
+  return constructor;


Please inline this check into the getPrefix check. That way the code should be simpler and pretty straight forward.

Yeah, thought about it. Will work on that and update the PR.

BridgeAR · 2018-08-25T16:16:29Z

lib/util.js

@@ -752,9 +784,11 @@ function formatValue(ctx, value, recurseTimes) {
      // Fast path for ArrayBuffer and SharedArrayBuffer.
      // Can't do the same for DataView because it has a non-primitive
      // .buffer property that we need to recurse for.
-      let prefix = getPrefix(constructor, tag);
+      const arrayType = isArrayBuffer(value) ? 'ArrayBuffer ' :
+        'SharedArrayBuffer ';


Due to the changes the types have to be 'ArrayBuffer' and 'SharedArrayBuffer' (without the whitespace at the end).

BridgeAR · 2018-08-25T16:16:45Z

test/parallel/test-util-inspect.js

-  [new SharedArrayBuffer(2), 'SharedArrayBuffer { byteLength: undefined }']
+   '[DataView: null prototype] {\n  byteLength: undefined,\n  ' +
+     'byteOffset: undefined,\n  buffer: undefined }'],
+  [new SharedArrayBuffer(2), '[SharedArrayBuffer : null prototype] ' +


See my comment above :-)

BridgeAR · 2018-08-25T16:17:19Z

lib/util.js

      if (prefix === '') {
-        prefix = isArrayBuffer(value) ? 'ArrayBuffer ' : 'SharedArrayBuffer ';
+        prefix = arrayType;
      }


This condition should now be obsolete.

BridgeAR · 2018-08-25T16:21:44Z

lib/util.js

    newVal = new clazz(value.length || 0);
  } else if (isTypedArray(value)) {
-    const clazz = findTypedConstructor(value) || Uint8Array;
+    let clazz;
+    if (Object.getPrototypeOf(value)) {


If the value has a prototype, we do not have to search for it anymore. Therefore it would be best to write this as:

const clazz = Object.getPrototypeOf(value) || findTypedConstructor(value) || clazzWithNullPrototype(Uint8Array, 'Uint8Array')

If I understand this correctly. If we write like the following:

const clazz = Object.getPrototypeOf(value) || findTypedConstructor(value) || clazzWithNullPrototype(Uint8Array, 'Uint8Array')

when there is no prototype, then findTypedConstructor will definitely return the typed array constructor. So we won't reach clazzWithNullPrototype (when there is no prototype) -- missing the subclass stuff.

You are correct, findTypedConstructor should always return the actual type.

Therefore it should likely be:

let clazz = Object.getPrototypeOf(value); if (!clazz) { const constructor = findTypedConstructor(value); clazz = clazzWithNullProrotype(constructor, constructor.name); }

BridgeAR · 2018-08-25T16:26:45Z

lib/util.js

+    }
+    return `[${fallback}: null prototype] `;
+  }
+
  if (constructor !== '') {


This condition tests for constructor !== '' so instead of testing in another if for the other case, we can just move the code above below this if. In that case we already know, that the constructor is an empty string and don't have to check again.

Well this is true, but the line:

return `[${fallback}: null prototype] `;

definitely needs a check whether a constructor is empty or not.

What I mean is:

if (constructor !== '') { ... as before ... } if (tag !== '' && fallback !== 'tag') { return `[${fallback}: null prototype] [${tag}] `; } return `[${fallback}: null prototype] `;

That way we are sure the constructor is definitely empty.

antsmartian · 2018-08-28T16:30:12Z

@BridgeAR Sorry to trouble you, may be you missed my last comments? :)

antsmartian · 2018-08-29T17:39:00Z

Ping @BridgeAR

BridgeAR

LGTM. Great work :-)

BridgeAR · 2018-08-29T17:53:13Z

lib/util.js

@@ -497,20 +497,19 @@ function getConstructorName(obj) {
 }

 function getPrefix(constructor, tag, fallback) {
+


Non-blocking nit: adding the line seems obsolete.

BridgeAR · 2018-08-29T17:55:32Z

lib/util.js

+// Creates a subclass and name
+// the constructor as `${clazz} : null prototype`
+function clazzWithNullPrototype(clazz, name) {
+


Non-blocking nit: I would remove the newline here.

BridgeAR · 2018-08-29T23:20:42Z

test/parallel/test-util-inspect.js

+   '[DataView: null prototype] {\n  byteLength: undefined,\n  ' +
+     'byteOffset: undefined,\n  buffer: undefined }'],
+  [new SharedArrayBuffer(2), '[SharedArrayBuffer: null prototype] ' +
+     '{ byteLength: undefined }']


Nit: this now seems to be a duplicated entry with the test below. Since the test below tests more than this one, it would be best to move this entry (or just keep it).

BridgeAR · 2018-08-29T23:22:40Z

CI https://ci.nodejs.org/job/node-test-pull-request/16867/

This makes sure the `null` prototype is always detected properly.

antsmartian · 2018-09-27T15:07:30Z

@BridgeAR I have moved the logic to internal/inspect.js. Please check and see if we can merge this.

antsmartian · 2018-10-01T09:41:03Z

@BridgeAR Anything else blocking this PR from getting merged?

This makes sure the `null` prototype is always detected properly. PR-URL: nodejs#22331 Fixes: nodejs#22141 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>

BridgeAR · 2018-10-02T16:27:13Z

Landed in f4e4ef5 🎉

@antsmartian thanks for sticking to it and being patient!

antsmartian · 2018-10-19T15:08:45Z

@BridgeAR Sorry to ping on the closed thread. If you remember, one of the early comments you mentioned that:

Would you please also add Arrays, Dataviews and ArrayBuffers? Errors, Dates and regular >expressions should be handled in a different PR as they do not show subclassing at all and that >should be fixed as well.

I was looking to implement for Error this morning. Lets say for this example:

class MyError extends Error {}
new MyError()

// want to print:
MyError
  at . . . 

//but currently prints:
Error:
  at

Also as said in the issue, we need to handle null cases as well (but that should be simple given we have getConstructorName and getPrefix in place). I have a made rough implementation:

diff --git a/lib/internal/util/inspect.js b/lib/internal/util/inspect.js
index 4cff372b33..c4e59116f1 100644
--- a/lib/internal/util/inspect.js
+++ b/lib/internal/util/inspect.js
 function formatError(value) {
+
+  // Preserve the sub-classing info
+  // in Error stack
+  function WrapError(error) {
+    const constructor = getConstructorName(value);
+    const prefix = getPrefix(constructor, '', 'Error').trim();
+
+    this.name = prefix;
+    this.message = error.message;
+
+    const stackStart = error.stack && value.stack.indexOf('\n    at');
+    if (error.stack && stackStart !== -1) {
+      let stack = error.stack;
+      stack = stack.slice(stackStart, stack.length);
+      stack = prefix + (this.message ? `: ${this.message}` : '') + stack;
+      this.stack = stack;
+    } else {
+      this.stack = error.stack;
+    }
+
+    return this;
+  }
+
+  // eslint-disable-next-line no-restricted-syntax
+  value = new  WrapError(value);
+
   return value.stack || errorToString(value);
 }

But looks like stack property shouldn't be mutated at all, just in case of message property was muted (test-internal-error.js does have a test for the same). I was wondering, how we can achieve that 🤔 Because when someone calls util.inspect with error object, I'm not sure how we can find if the message property is muted or not.

BridgeAR · 2018-10-19T15:32:24Z

@antsmartian feel free to chat with me on IRC (even though I am not always around).

It is indeed not a good idea to mutate the input at all. Instead, we should be able to just create the output as we expect it to look like and we only change to change a single string: the stack after accessing it.

We also already have the constructor name and don't have to get it again.

So it should be similar to:

function formatError(value, constructorName) {
  if (value.stack) {
    let stack = value.stack;
    if (!stack.startsWith(constructorName)) { // Note: this check is likely not sufficient.
      return stack.replace(..., constructorName);
    }
    return stack;
  }
  return errorToString(value);
}

antsmartian · 2018-10-19T15:47:20Z

@BridgeAR Thanks. (can you just tell me your IRC handle, I searched for bridgear, doesn't look like its correct)

This makes sure the prototype is always detected properly. PR-URL: nodejs#22331 Fixes: nodejs#22141 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>

This makes sure the prototype is always detected properly. Backport-PR-URL: #23655 PR-URL: #22331 Fixes: #22141 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: John-David Dalton <john.david.dalton@gmail.com>

nodejs-github-bot added the util Issues and PRs related to the built-in util module. label Aug 15, 2018

antsmartian commented Aug 15, 2018

View reviewed changes

antsmartian changed the title ~~util: Handle null prototype on inspecting~~ util: Handle null prototype on inspect Aug 15, 2018

jasnell requested a review from BridgeAR August 15, 2018 14:08

BridgeAR reviewed Aug 15, 2018

View reviewed changes

TimothyGu added the wip Issues and PRs that are still a work in progress. label Aug 23, 2018

antsmartian force-pushed the util_proto branch 2 times, most recently from b6fccaa to 7ddca64 Compare August 25, 2018 11:36

antsmartian force-pushed the util_proto branch from 7ddca64 to cd98c73 Compare August 25, 2018 11:48

BridgeAR reviewed Aug 25, 2018

View reviewed changes

antsmartian force-pushed the util_proto branch from cd98c73 to 09c796c Compare August 25, 2018 15:57

BridgeAR reviewed Aug 25, 2018

View reviewed changes

antsmartian force-pushed the util_proto branch from 09c796c to 9dc530c Compare August 28, 2018 17:45

BridgeAR approved these changes Aug 29, 2018

View reviewed changes

BridgeAR removed the wip Issues and PRs that are still a work in progress. label Aug 29, 2018

antsmartian force-pushed the util_proto branch from 9dc530c to 9112c00 Compare August 30, 2018 14:53

antsmartian and others added 3 commits September 27, 2018 19:42

util: Handle null prototype on inspect

96fdc40

util: improve prototype detection in .inspect()

d14c3e6

This makes sure the `null` prototype is always detected properly.

util: Move handling null logic to inspect file

215700c

antsmartian force-pushed the util_proto branch from 4f52f0f to 215700c Compare September 27, 2018 14:45

BridgeAR approved these changes Sep 27, 2018

View reviewed changes

BridgeAR closed this Oct 2, 2018

targos added the backport-requested-v10.x label Oct 3, 2018

antsmartian deleted the util_proto branch October 17, 2018 17:17

MylesBorins added backported-to-v10.x and removed backport-requested-v10.x labels Nov 12, 2018

codebytere mentioned this pull request Nov 27, 2018

v10.13.1 proposal #24675

Closed

codebytere mentioned this pull request Nov 29, 2018

v10.14.2 proposal #24727

Merged

		@@ -497,20 +497,19 @@ function getConstructorName(obj) {
		}

		function getPrefix(constructor, tag, fallback) {

util: Handle null prototype on inspect #22331

util: Handle null prototype on inspect #22331

Conversation

antsmartian commented Aug 15, 2018 • edited Loading

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BridgeAR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TimothyGu commented Aug 16, 2018

antsmartian commented Aug 16, 2018 • edited Loading

devsnek commented Aug 16, 2018

antsmartian commented Aug 16, 2018

antsmartian commented Aug 16, 2018

BridgeAR commented Aug 17, 2018

TimothyGu commented Aug 23, 2018

BridgeAR commented Aug 23, 2018

TimothyGu commented Aug 23, 2018

antsmartian commented Aug 23, 2018

antsmartian commented Aug 25, 2018 • edited Loading

BridgeAR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antsmartian Aug 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antsmartian commented Aug 28, 2018

antsmartian commented Aug 29, 2018

BridgeAR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BridgeAR commented Aug 29, 2018

antsmartian commented Sep 27, 2018

antsmartian commented Oct 1, 2018

BridgeAR commented Oct 2, 2018

antsmartian commented Oct 19, 2018

BridgeAR commented Oct 19, 2018

antsmartian commented Oct 19, 2018 • edited Loading

antsmartian commented Aug 15, 2018 •

edited

Loading

antsmartian commented Aug 16, 2018 •

edited

Loading

antsmartian commented Aug 25, 2018 •

edited

Loading

antsmartian Aug 25, 2018 •

edited

Loading

antsmartian commented Oct 19, 2018 •

edited

Loading