Enable circular references when serializing JSON

[WebReflection]

As previously asked on circular-json repository,
this PR would like to bring the heavily battle tested
circular-json module as parser, enabling circular references
to travel over the socket.

Thanks for considering this change.

[lopugit]

Sooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo beautiful, do you have a btc wallet?

I implemented it, working like a charm

[lopugit]

I think that the hasBinary module isn't compatable, you get a call stack error because of this function

function hasBinary (obj) {
  if (!obj || typeof obj !== 'object') {
    return false;
  }

  if (isArray(obj)) {
    for (var i = 0, l = obj.length; i < l; i++) {
      if (hasBinary(obj[i])) {
        return true;
      }
    }
    return false;
  }

  if ((typeof global.Buffer === 'function' && global.Buffer.isBuffer && global.Buffer.isBuffer(obj)) ||
     (typeof global.ArrayBuffer === 'function' && obj instanceof ArrayBuffer) ||
     (withNativeBlob && obj instanceof Blob) ||
     (withNativeFile && obj instanceof File)
    ) {
    return true;
  }

https://github.com/socketio/has-binary/blob/master/index.js

I'm not smart enough to figure out how to solve it.... you'd either need to taint... well actually I'm sure you can just implement whatever method you use in CircularJSON to identify a circular reference.

How do you do it actually, tainting orr?

[lopugit]

I can't figure out your code, you should comment each function a little more, some parameter notes

function stringifyRecursion(value, replacer, space, doNotResolve) {
    /** 
     @param value what is value, bit of a bio
     @param replacer what is replace, bit of a bio
     etc..
    */
    return JSON.stringify(value, generateReplacer(value, replacer, !doNotResolve), space);
}

Idk if you care but that's how I comment my functions, I don't do it for everything and I hated the amount of commenting asked for at uni but, I comment some things now, bio's of variables basically, it makes understanding a function quite easy

[WebReflection]

do you have a btc wallet?

yes, but no idea how to use it and apparently no online service from bitcoin .com. hints welcome, or I can send you the qr code if you want, dunno, really.

I think that the hasBinary module isn't compatable, you get a call stack error because of this function

can you please provide the bare minimum code to fail ?

I can't figure out your code, you should comment each function a little more

it's an old project that worked well so far, I didn't need to maintain it much and quite possibly I don't need to maintain it even in this case but I need a code to reproduce the issue to understand what is it.

Thanks

[lopugit]

          let child = {
            foo: 'bar'
          }
          let parent = {
            zoo: 'gar',
            child
          }
          child.parent = parent
          this.$socket.emit('test', child)

Will give the error

[WebReflection]

heh ... that means that hasBinary is incompatible with recursive scans. No need to change anything in CircularJSON, will PR a change for hasBinary too.

[lopugit]

Also, either create a localbitcoins.com account, or download electrum or jaxx, and you'll be able to see your wallet address under you profile, I will send you a little tip, I would recommend using electrum or jaxx, then you can just store the seed in the cloud.. encrypted if you use openssl or gpg, and you'll have a multi platform wallet that you can access anywhere, and just store bitcoins on, the tip I send might grow 200% in the next year, maybe more ;)

yes it's hasBinary's fault

[WebReflection]

@lopugit let's take the tip thing offline ( my name dot my surname at gmail dot com )

I have PR'd changes with tests for has-binary. Is there any other place I need to make recursively scan-safe?

[lopugit]

I think you changed hasBinary instead of hasBinary2 :')

emailed

[WebReflection]

you linked me this one ... https://github.com/socketio/has-binary/blob/master/index.js

[lopugit]

I'm sorryyyyyyyyyyyyy I just google hasBinary and it came up with the socket-io repo..... have I mentioned I'm a bit retarded... :') sorry, I hope it's not much of a change

[WebReflection]

I have no idea where the hell is has-binary2 so I've filed randomly a PR to the only thing that comes up looking for has-binary2 on Google

[WebReflection]

@darrachequesne where is the current has-binary2 repository to send a PR ? Thanks

[WebReflection]

... I hope I've found it: darrachequesne/has-binary#3

[lopugit]

I'm confused about how to clone this repo?

https://github.com/WebReflection/volumio-plugins/blob/57ac90a94d2377522a8f324d115a082b7d769ef3/plugins/miscellanea/rfampswitch/node_modules/has-binary2/index.js

The tree has a bunch of letters and numbers, but the branch says gh-pages, but when I checkout gh-pages, there's no misc plugin in the plugins folder. I don't understand.

But it looks like that's the repo, looks like it anyway

[WebReflection]

it's not, I've found it. We should just stop and wait now, or they'll close this PR because of the mess in it.

[darrachequesne]

Hi! Thanks for the pull request.

I don't think this should be added in the default parser, since the circular-json would increase the size of the client build, while not being needed in most cases. How about creating a custom parser?

[flotwig]

For anyone reading this thread in the future looking for a way to use this code, I've taken @WebReflection's fork and added his modified has-binary2 to it and published it on NPM. So if you need to use circular structures in your JSON, now you can just npm install socket.io-circular-parser

https://github.com/cypress-io/socket.io-circular-parser

[WebReflection]

@flotwig before this get noticed, I suggest you switch to flatted instead, or even use pocket.io with flatted as parser 👋

[flotwig]

@WebReflection I tried to drop-in flatted as a replacement, but it was giving me some strange error (sorry, not at work computer rn), I'll give it another try later.

Re: pocket.io - it doesn't look like it supports custom agents, which is a requirement for our code base. Looks cool though! I'm always a fan of decreasing complexity.

[WebReflection]

it doesn't look like it supports custom agents

you are right, although about flatted, it passes all tests CircularJSON does, but it flattens out the result, it doesn't preserve the structure once serialized (it does once deserialized), although trusting CircularJSON deserialization without using CIrcularJSON is a foot gun prone approach, which is why I've created flatted, so it's clear not everyone can change the value returned by flatted, and only flatted can deserialize its output (or any parser that implements the same 20 LOC strategy).

Anyway, do what's best for your code base, I was just providing updated feedbacks on the circuoar-json lib 👋

[lopugit]

@WebReflection @flotwig very nice work fellas

I've been caught out by flatted's new approach of turning objects into arrays, is it really necessary, is it more efficient or something or? cause it's almost like compressing.

Or does it prevent JSON.parsing a CircularJSON/flatted.stringified object?

[WebReflection]

You cannot parse other outcome, and that has never been the case, even with circular json, it was possible by accident, and it was disaster prone.

The output is also an implementation detail you shouldn't care about, like a binary format.

[dustingraham]

Issue is that between has-binary2 and socket.io, accidentally sending the wrong object to emit can cause significant confusion. I adjusted a function signature, which caused a circular object to be passed to emit rather than a string.

Even using a debugger and breakpoints and stepping through the issue it was not easily apparent what/why it was throwing a stack overflow. I thought it was due to some setTimeout timers I had and spent quite a lot of time diagnosing this. I probably won't forget this issue again, but could save others time.