Support Web APIs.

[hongzhidao]

@xeioex @drsm

Web APIs is an alternative usage.
https://developer.mozilla.org/en-US/docs/Web/API/Response/Response
It will behave in njs like below.

function hello(request) {
      return new Response('hello');
}

function subrequest(request) {
      var sr = new Request('/sub');
      sr.fetch()
      .then(response) {
          return response;
      };
}

In the beginning, njs has individual response object. foo(req, res). After that, res is merged into req, so we have to rename some methods like headersIn, headersOut.
If we introduce these things, Request, Response, Headers, Body, I believe it can be easier to use, and we just implement the standard methods with them.
The second change is that the subrequest is also easier to use. Indeed, users need not know what subrequest is, they just want to send a request. (But the implementation will be upon on nginx subrequest).
Still, further, it can be welcome after async, await are introduced.

Highly welcome to your suggestions. I'd like to try it without changing current usage by put them in precontent phase.

I'm still not familiar with async, await, Promise, it's good to have a better example of how to write with subrequest below. @drsm

BTW, do you know whether the above objects are standard belong to js?
I'm wondering which software has already implemented them, where are they used?

[xeioex]

@hongzhidao

BTW, do you know whether the above objects are standard belong to js?

There are a ton of web standards. ECMAScript® 2020 Language Specification (JavaScript) is just one of the many. DOM is another example. They are related but otherwise orthogonal.

I'm wondering which software has already implemented them, where are they used?

https://developer.mozilla.org/en-US/docs/Web/API/Response/Response#Browser_compatibility

BTW, every feature is marked as "Experimental. Expect behavior to change in the future."

So we probably should wait a bit until dust is settled.

I'm still not familiar with async, await, Promise, it's good to have a better example of how to write with subrequest below.

Take a look at an example.

[hongzhidao]

@xeioex @drsm @lexborisov

Take a look at an example of generator.

function* genFunc() {
    yield 'a';
    yield 'b';
    yield 'c';
}

genObj = genFunc();

console.log(genObj.next());
console.log(genObj.next());
console.log(genObj.next());
console.log(genObj.next());

We've talked about the usage of lua-module, that developers can write async-operations in sync way.

local res = ngx.location.capture("/some_other_location")
if res then
  ngx.say("status: ", res.status)
  ngx.say("body:")
  ngx.print(res.body)
end

Lua has a mechanism called coroutine. See an example.

thread2 = coroutine.create(function()
  for n = 1, 5 do
    print("honk "..n)
    coroutine.yield()
  end
end)

coroutine.resume(thread2)
-->> honk 1
coroutine.resume(thread2)
-->> honk 2

Actually, each request is wrapped as a coroutine function, so lua-module can do the async things like capture.

The purpose of generator and coroutine are the same. Compare to normal functions that have to run code from start to end, these functions can pause in any place.

1. Iterator and Generator
   I suggest we don't need to spend more time on improving the usage of async-operations in nginx module, now it is subrequest since there are a few different ways.
   (callback, promise, async/await, generator).
   It's better to implement that util these syntaxes are finished. Now, promise way looks OK.

What about starting to implement iterator and generator?
If nobody is on it, I plan to do it. Welcome to your suggestions.

2. parser refactoring
   BTW, I'm wondering it's ready to have a look at this patch? @lexborisov

[drsm]

@hongzhidao

Hi!

What about starting to implement iterator and generator?
If nobody is on it, I plan to do it. Welcome to your suggestions.

Take a look at this example.
I think, the main problem with iterator protocol will be a large number of temporaries generated.
It is unusable without GC.

[xeioex]

Among Web APIs, the most interesting and promising is Web Crypto API.

[hongzhidao]

Among Web APIs, the most interesting and promising is Web Crypto API.

1. It seems it's better to put these features into another place like njs-lib.c and njs-lib.h.
2. Do you know what library is good to use for implementing Crypto? (OpenSSL?)

[xeioex]

@hongzhidao

Do you know what library is good to use for implementing Crypto? (OpenSSL?)

yes, ideally we want to use the same OpenSSL-like library which nginx is using. The trick is to get the right library (including the the libs that were provided as --with-openssl=...)

As on now njs configure does not see this information when it is built as a part of nginx (https://github.com/nginx/njs/blob/master/nginx/config.make).

[hongzhidao]

@xeioex

Just take a look at the implementation of Lua. lcrypto
BTW, I'm thinking that some features like Web APIs URL, Request, etc even repl can be implemented with js since it needs more time by c way. url

[xeioex]

@drsm

I am thinking about the way to split-apart byte strings and ordinary strings (http://nginx.org/en/docs/njs/reference.html#string),
because it is became increasingly complicated to handle them as a valid variants of a String. The similar issue with str vs bytes in python (https://stackoverflow.com/questions/18034272/python-str-vs-unicode-types).

Byte-string - a sequence of bytes.
Ordinary strings - a unicode text string, internally represented as a valid UTF-8 string.

Historically byte string were introduced as as thin (non-copy) wrappers around nginx native objects (for example r.requestBody, r.variables.hostname) and we want this property to stay.
We have a second requirement, all internally created strings are always a valid UTF-8 strings.
Currently the only way to inject byte-strings is only through nginx modules.

The problems arise when we work with byte-strings internally (concating ordinary and byte strings, joining array which contains byte strings), when byte-string cannot be represented as a valid UTF-8 string (for example a byte '0x9d' is not a valid UTF-8).

There is also a backward compatibility issue, we should not change the current API too much.

For example:

1. r.requestBody - what type should it be? currently it is a byte-string (consider JSON.stringify(r.requestBody) when r.requestBody is invalid UTF-8 string).
2. r.variables - are also byte strings, because there are binary variables ($binary_remote_addr)
3. s.on('upload', function (data, flags) {}) - data is also a byte string.
   ...

Ideally we want to get rid of byte-string altogether (replacing them with TypedArray).
But there are API issues and performance issues.

1. TypedArray API is too different from String API, so a lot of existing code have to changes.

I am thinking about some kind of Proxy object (similar to Blob).

 NjsBlob 
   NjsBlob.prototype.arrayBuffer() 
   NjsBlob.prototype.text() -> convert to UTF8 string()
   NjsBlob.prototype.toString() -> text() // implicit conversion to String, but exception when it is not possible

 JSON.stringify(r.requestBody) // works out of the box

  r.requestBody.split('&') // requires changes, but seems manageable ->  r.requestBody.text().split('&')

NjsBlob != Blob - because the latter returns Promise(), seems too much overhead for simple things

Possibly adding later StringView.

What is your take on it?

[drsm]

@xeioex

I like a NjsBlob solution.
Some thoughts:

1. typeof r.requestBody == 'object'. I think it is better to freeze that object, as well as it's prototype, so the things can be optimized internally.
2. r.requestBody + r.requestBody will be a UTF-8 string, so it's a breaking change. I think we need NjsBlob.from() and NjsBlob.concat() like nodejs Buffer does.

[xeioex]

@drsm

r.requestBody + r.requestBody will be a UTF-8 string, so it's a breaking change.

Yes, maybe implicit toString() is not a good idea.

> var a = new Uint8Array([0x41, 0x42]); var b = new Blob([a.buffer]); await b.text()
"AB"
> b.toString()
"[object Blob]"

I think we need NjsBlob.from() and NjsBlob.concat() like nodejs Buffer does.

Buffer is not a standard, ES6 version looks like:

const a = new Int8Array( [ 1, 2, 3 ] )
const b = new Int8Array( [ 4, 5, 6 ] )
const c = Int8Array.from([...a, ...b])

[drsm]

@xeioex

r.requestBody + r.requestBody will be a UTF-8 string, so it's a breaking change.

Yes, maybe implicit toString() is not a good idea.

Looks so, the problem that it will partitialy work.

Buffer is not a standard,

Yes, but it's well-known and it is a Uint8Array subclass in modern versions.
And if we're dealing with node modules, browserify will bring its own implementation anyway.

[drsm]

@xeioex

 NjsBlob 
   NjsBlob.prototype.arrayBuffer() 
   NjsBlob.prototype.text() -> convert to UTF8 string()

Maybe is better to use properties instead?

NjsBlob 
    NjsBlob.prototype.size 
    NjsBlob.prototype.arrayBuffer 
    NjsBlob.prototype.text -> create/return a `string` representation

[xeioex]

@drsm

Maybe is better to use properties instead?

Yes, am considering it.

[xeioex]

@drsm

Meanwhile I am studying the way to port existing njs code automatically, it seems it is pretty straightforward in JS world.

For example, to robustly rename in the code String.bytesFrom() -> String.from():

cat example.js
function jwt(data) {
    return data.split('.').map(v=>String.
                           bytesFrom(v, 'base64url'));
}

$ jscodeshift -p -d -t njs-0.5.0-api-transform.js example.js 
Processing 1 files... 
Spawning 1 workers...
Running in dry mode, no files will be written! 
Sending 1 files to free worker...

The njs API transformer will do the following replacements:
Core:
        String.bytesFrom() -> String.from()

function jwt(data) {
    return data.split('.').map(v=>String.
                           from(v, 'base64url'));
}

All done. 
Results: 
0 errors
0 unmodified
0 skipped
1 ok
Time elapsed: 0.456seconds 

$ cat njs-0.5.0-api-transform.js
function transformer(file, api, options) {
  const j = api.jscodeshift;
  const source = j(file.source);


  console.log(`
The njs API transformer will do the following replacements:
Core:
	String.bytesFrom() -> String.from()

`);

  /*
   * String.bytesFrom() -> String.from();
   */
  source
    .find(j.CallExpression, {
      callee: {
        type: 'MemberExpression',
        object: {
          name: 'String'
        },
        property: {
          name: 'bytesFrom'
        }
      }
    })
    .replaceWith(path => { path.value.callee.property.name = 'from'; return path.value;});

  return source.toSource();
}

module.exports = transformer;