Access to Browser API webRequest

[vyach-vasiliev]

Is it possible to make access to Browser API from the User-Script code? At least for Chrome-like browser.

Sometimes this really is really not enough and I do not want to write everytime extra extensions for the sake of a couple of lines of code.

Similar to:

    var filter = {urls: ["http://*/*", "https://*/*"], tabId: currentTabId };
    var opt_extraInfoSpec = ['blocking'];
    GM_webRequest.onBeforeRequest.addListener(
        callback, filter, opt_extraInfoSpec);

[tophf]

The thing is, blocking code is synchronous so it must be inside the background page of Tampermonkey and you surely understand it would be insane to allow arbitrary userscripts to run there.

A viable solution might be declarative syntax e.g. provide an array of rules that specify what to do.

[tophf]

On the other hand, maybe Tampermonkey's sandbox can be applied to userscript code in the background page too...

[vyach-vasiliev]

@tophf I understand it and certainly agree with you what is required declarative syntax for rules. If I knew exactly how Tampermonkey works, I would try to implement it.

Agree that this option would give a strong advantage before similar extensions?

[derjanb]

Ok, I've created a preview that shows how GM_webRequest and @webRequest could possibly work.

After downloading and drag and dropping it to the extension page you can install this script:

// ==UserScript==
// @name         GM_webRequest testing
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        *://*/*
// @include      http://new_static.url/*
// @include      *//redirected.to/*
// @grant        GM_webRequest
// @webRequest   [{"selector":"*cancel.me/*","action":"cancel"},{"selector":{"include":"*","exclude":"http://exclude.me/*"},"action":{"redirect":"http://new_static.url"}},{"selector":{"match":"*://match.me/*"},"action":{"redirect":{"from":"([^:]+)://match.me/(.*)","to":"$1://redirected.to/$2"}}}]
// ==/UserScript==

var currently_active_webrequest_rule = JSON.stringify(GM_info.script.webRequest); // == @webRequst header from above

GM_webRequest([
    { selector: '*cancel.me/*', action: 'cancel' },
    { selector: { include: '*', exclude: 'http://exclude.me/*' }, action: { redirect: 'http://new_static.url' } },
    { selector: { match: '*://match.me/*' }, action: { redirect: { from: '([^:]+)://match.me/(.*)',  to: '$1://redirected.to/$2' } } }
], function(info, message, details) {
    console.log(info, message, details);
});

/*
  Notes:
    Action:
      The final redirect: URL needs to be included into the scripts @match or @include header
    Selector:
      If just a string is given, it is interpreted as include: property.
      include:, exclude: and match: properties are supported and can have an array or a string value - the syntax equals @include, @match and @exclude
    @webRequest may have a human readable format in the future, but for now it's the stringified first argument of GM_webRequest and allow request 
      manipulation even if the script didn't run yet. The only downside atm is that you need to re-register the rules (which overrides all previous ones)
      in order to get manipulation events. GM_webRequest(currently_active_webrequest_rule, function...)
*/

This all is not heavily tested so expect bugs here and there. :)

[jspenguin2017]

@derjanb
Thanks! This is going to help a lot!

If I understood right, the callback function will only receive information about the WebRequest once it is done and is not able to manipulate it.
In this case, I think being able to redirect to a generated string as the response content would be helpful.

I know it is not a good idea to let random function to run in privileged extension context, but there can be times when that would be handy. What is your opinion on this? Should I just create another extension to do this or we can have a special header for it?
The redirection and blocking that the design above allows can mostly be done by uBO, Adguard, or some other ad blockers, what they cannot do is run complex function to make dynamic decision.

Also, as a side thought, shouldn't it be named TM_webRequest? Since this functionality is obviously not in Greasemonkey.

[vyach-vasiliev]

@derjanb
Wow, thank you for such a prompt reply! ⚡

I try to play with https://ssl.gstatic.com/gb/images/v1_b3735dd8.png, but not one of the rules does not work for me.
What am I doing wrong?

// ==UserScript==
// @name         GM_webRequest testing
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the world!
// @author       You
// @match        *://*/*
// @include      *gstatic.com/*
// @include      https://google.com/*
// @include      *//google.com/*
// @grant        GM_webRequest
// @webRequest   [{"selector":"*cancel.me/*","action":"cancel"},{"selector":{"include":"*","exclude":"http://exclude.me/*"},"action":{"redirect":"http://new_static.url"}},{"selector":{"match":"*://match.me/*"},"action":{"redirect":{"from":"([^:]+)://match.me/(.*)","to":"$1://redirected.to/$2"}}}]
// ==/UserScript==

console.log("GM_webRequest start");

var currently_active_webrequest_rule = JSON.stringify(GM_info.script.webRequest); // == @webRequst header from above

GM_webRequest([
    //{ selector: '*gstatic.com/*', action: 'cancel' },
    { selector: { include: '*gstatic.com/*', exclude: 'http://exclude.me/*' }, action: { redirect: 'https://google.com/' } },
    //{ selector: { match: '*://*.gstatic.com*' }, action: { redirect: { from: '([^:]+)://(.+)gstatic.com/(.*)',  to: '$1://google.com/$2' } } }
], function(info, message, details) {
    console.log(info, message, details); // This also does not work
});

[derjanb]

Ah sorry, I forgot to add this information. Since intercepting requests makes things slower Tampermonkey only handles the following request types at the moment: 'sub_frame', 'script', 'xmlhttprequest' and 'websocket'.

All other are considered to be replaceable by a userscript, even after they were loaded, but that's discussable.

[vyach-vasiliev]

Hm, can be done something like?

/* here, any calculations of javasscript are not related to the page (setting vars, functions) */
GM_webRequest([
        { selector: '*gstatic.com/*', action: 'replace' },
], function(info, message, details) {
        console.log(info, message, details);
	details.url = details.url.replace('gstatic', 'gstatic-2');
	return details;
});

input  >>> https://gstatic.com/index.php?foo=bar...
output >>> https://gstatic-2.com/index.php?foo=bar...

We change url or body of the request and query data from the site.
With support for GET and POST requests.
Of course, according to the rules, the script code will be executed once for each request.

[derjanb]

We change url or body of the request and query data from the site.

Atm only the URL via:

{ selector: { match: '*://match.me/*' }, action: { redirect: { from: '([^:]+)://match.me/(.*)',  to: '$1://redirected.to/$2' } } }

With support for GET and POST requests.

Shouldn't make a difference.

details.url = details.url.replace('gstatic', 'gstatic-2');

This can't work because all communication is asynchronous. This means when the request is inspected by the background page, then there is no synchronous way to contact the userscript to make a decision. That's why all rules need to be defined when the requests happens.

[vyach-vasiliev]

@derjanb
You did not understand me. I meant, this scheme:

1. Script installation
2. Loading the script into the storage (function => toString)
3. Reload background page and load Userscripts with GM_webRequest into array at background page.
4. Start load request (and check rules Userscripts)
5. Loading a script (string) from the storage(array) and executing into callback of webRequest

So will it work?
I don't understand webRequest's work well, but I'm afraid to give up hope for success 😃

[derjanb]

Loading a script (string) from the storage(array) and executing into callback of webRequest

The background page has all extension permissions. It's not possible to run foreign JavaScript code there without leaking these permissions. Sorry. That's why a defined rule set is needed to tell the background page what needs to be done with a request.

[vyach-vasiliev]

@derjanb
Oh, thanks now everything is clear. But the parameters can still be read?
There can be such a parser?

var rules = [ 
        { event: 'request', key: 'url', action: replace: { from: '([^:]+)://match.me/(.*)', to: '$1://redirected.to/$2'}, },
];  // Execution in order
GM_webRequest(rules, function(info, message, events) {
        console.log(info, message, events); // ex. events.request.url or events.response.body
}); // Reload background page and load rules with GM_webRequest into array rules at background page.

Help:

• actions (action)
  • block
  • replace
• events (event)
  • request
  • response
• event keys (key)
  • url
  • header
  • body
• types request (type, default - all types)
  • xmlhttprequest or fetch
  • window location
• methods request (method, default - all methods)
  • POST
  • GET
  • etc

UPDATE 03.06.17

[LennyPenny]

I don't think only intercepting certain request types is a good idea. There might be instances where one wouldn't want some request to even be made in the first place. Also in general replacing things that already have been loaded by something that too has to first be loaded seems pretty slow as well.

How about being able to configure which requests types will be intercepted?

[vyach-vasiliev]

@LennyPenny You mean, XHR and Location types of query handlers, as well as the query types like POST, GET and others?
Give an example please, if it is not difficult for you 😃