Skip to content
This repository has been archived by the owner on Oct 16, 2020. It is now read-only.

Chrome Manifest V3 discussion megathread #238

Open
mikhoul opened this issue Jan 21, 2019 · 104 comments
Open

Chrome Manifest V3 discussion megathread #238

mikhoul opened this issue Jan 21, 2019 · 104 comments
Labels

Comments

@mikhoul
Copy link

mikhoul commented Jan 21, 2019

Hi,

I really like Nano more than UBO but I was asking to myself what will be your direction if the proposal is adopted and that the actual API is neutered ?

My understanding is that Chrome (for me ) would become inferior to Firefox for adblockers (unless Firefox implement the same restriction which I doubt).

If the change happen they can happen in the next 12 months which is relatively fast so the solutions/alternatives I've see are:

  • Make an official version for Firefox and drop Chrome.
  • Make an Edge version with their new browser with blink engine IF they don't have the same restriction.
  • Submit to the new API and have an inefficient adblocker comparing to today.
  • Forking Chromium keeping the old API as it is today and have a version especially build for Nano.

Maybe there is other solutions that I don't foresee. 🤔

What is your take on this @jspenguin2017 ?

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 22, 2019

I see 3 options:

  • Use the new API
  • Fork Chromium
  • Make a native app

I'll explain in detail later.

I would guess Edge will closely follow Chromium, so that's probably not going to work.
What about Firefox? Well, I'll explain later.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 22, 2019

Option 1: Use the new API

If the new API is really weak, then I don't think Firefox can survive for long. If websites can more easily get around the weak APIs of Chrome, then website owners will have a motivation to push everyone to use Chrome. And soon Chromium-based browsers will probably become the only usable browsers.

If the new API is OK, then converting to the new API is probably going to be the easiest solution.

The currently proposed APIs are on the weak side. If the 30k limit is lifted and rules can be dynamically modified, then I would say it's OK.

In the other thread, gorhill listed his concerns:

  • Rules count limit
  • Dynamic filtering
  • The important option
  • The csp and inline-script options
  • Extended syntax

The rules count limit is a legitimate concern (one of my two primary concerns mentioned above), as EasyList alone has more than 80k rules.

As for dynamic filtering, I never use it. To be honest I don't even know how it works. So as far as I'm concerned, that's not an issue.

When it comes to the important option, I think it's possible to implement with more than one extension. Each filter becomes an extension, one extension to enforce all important rules, and one master extension to help with whitelisting and management.
Note that filters maintainers will probably have to publish their own filters as extensions then attach them to the public APIs of the coordinator. It's not going to be pretty, but theoretically possible.

For the csp option, I think there'll be some way left to modify headers. If not, it can still be implemented using content scripts (script snippets). The inline-script option can be (AFAIK, is) implemented using csp.

For extended syntax, these are a few notable ones:

  • Cosmetic filtering
  • Script snippets injection
  • Request redirection
  • The badfilter option
  • The important, csp, and inline-script options (covered above)

For cosmetic filtering (normal and extended), pretty much everything is implemented in content scripts, which are not part of the manifest v3 proposal, so I would assume they are not going to change.

The new APIs will actually make script snippets injection better, as the proposal contains new APIs that allow extensions to dynamically modify content scripts.

Same thing for redirection, (the current draft of) declarativeNetRequest supports redirect rules natively.

The badfilter option is resolved in compile time, so it doesn't matter.

The only thing missing is RegExp rules (and HTML filtering if you want to count that). I don't think it'll be a problem because neither uAssets (nor me) really use RegExp rules. On the other hand, overly broad RegExp rules from EasyList are constantly causing problems.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 22, 2019

Option 2: Fork Chromium

I would assume code for manifest v2 will eventually be removed, so the forking process will eventually become not a simple flag flip. Maintaining a browser fork is really difficult in general, and vulnerabilities can be opened if not careful.

Another way is to use Electron to build a new browser. Unlike WebExtension, Electron provides really powerful APIs, which are unlikely to go away. However, the amount of work to maintain a browser is likely more than that of a fork, but since Electron apps use JavaScript and not native code, vulnerabilities are less likely to occur.

Also, implementing WideVine (DRM) will be really difficult in either case.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 22, 2019

Option 3: Make a native app

There's a shell there's a way. I'm sad to see that there's no open source native app that does more than DNS blocking.

A native app is the best way to get around the constantly changing browser APIs, and as a bonus, it'll probably work for all browsers and even non-browser apps.

I'm not sure how hard this will be though, last time I tried it didn't go so well.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 22, 2019

I probably will go with option 1 or 3. As for the timeline, I don't think it'll happen within 12 months. Right now, the APIs are not released yet, and the phase out period will likely take more than a year.

I think it's a good time to get started on option 3, if things go wrong, option 1 can be used as fallback.

@mikhoul
Copy link
Author

mikhoul commented Jan 22, 2019

First thanks for your very detailed answer. 👍

The problem I see with option 3 if my understanding is fine is that a native app would not be able to remove some objects like cosmetic filters are doing ?

For me a native app sound like when I use AdAway on android, I'm pretty sure I'm missing something here about how would work a native app 🤔 ❓

I would miss "Dynamic filtering" since I use it occasionally to un-break some filters that are good but that on specific website cause problems.

I don't think Firefox will disappear, it can lose users but Google will always keep Firefox alive as a caution to prevent a split of their business by US Gov (like Microsoft have done with Apple).

IMO if Google weaken their API for blocking ads it could revigorante Firefox with power-users and gain 2-3% of users. When I migrated from Firefox to Chromium I dragged with me ~150 users and those users today recommend Chrome/chromium to their friends and I'm pretty sure I'm not alone.

But Mozilla management is crazy those day they removed so many things/features that I migrated to Chromium to stop from suffering from the many memory leaks in Firefox. So in one way it would not surprise me that they do some crazy thing in the future like following the very same Chrome API unless the management at Mozilla change.

For the fork I was thinking it was more easy than you said it is to keep the old API, like some module you glue to the main code when you build chromium and that don't require lot of maintenance. I've seen Slimjet a Chromium fork and with an integrated ad-blocker so I was thinking it was "easy" to integrate in a fork.

Time will tell. :octocat:

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 22, 2019

native app would not be able to remove some objects like cosmetic filters are doing

Again, there's a shell there's a way. Native apps are (theoretically) able to do cosmetic filtering on their own as they can patch everything the browser receives, however, I do agree that for cosmetic filtering, it's easier to implement using extensions.

There's nothing stopping native apps from loading extensions into browsers, so if needed, the native app can have a companion extension.

sound like when I use AdAway on android

AdAway is a DNS manager, it is really limited in what it can do. I'm talking about native apps that can repack HTTPS requests.

but Google will always keep Firefox alive as a caution to prevent a split of their business by US Gov

Possibly.

But Google can also point to those 20+ Chromium forks and say their code is under a permissive license, and everyone is free to fork and compete.

For the fork I was thinking it was more easy than you said

I never tried so I don't know for sure. Maybe you're right.
If you have more experience with the Chromium code base maybe you can start a Chromium fork?

@jspenguin2017
Copy link
Member

Actually instead of trying to reinventing the wheel and fail miserably, I'll probably just take a wheel off the shelf.

I'll probably still need a JavaScript runtime though, as RegExp in filters are JS RegExp. For a prototype it'll be fine, if it works out I'll worry about optimization later.

@mikhoul
Copy link
Author

mikhoul commented Jan 23, 2019

Thanks for your complete and detailed answer it's enlighting and it remove interrogations I had. 👍

So it would be a little bit how AdGuard work, I never used AdGuard but my understanding is that there is a component that work at the OS level and another part as an extension for better efficiency.

If you have more experience with the Chromium code base maybe you can start a Chromium fork?

I don't have experience but it is something I 'm thinking about for 2-3 months to be able to modify the Chromium GUI and keep it tight, maybe I will give it a try this year, this issue will ad some motivation make this project higher in my priority.

Regards :octocat:

@onmyouji
Copy link

onmyouji commented Jan 23, 2019

Another way is to use Electron to build a new browser. Unlike WebExtension, Electron provides really powerful APIs, which are unlikely to go away. However, the amount of work to maintain a browser is likely more than that of a fork, but since Electron apps use JavaScript and not native code, vulnerabilities are less likely to occur.

I don't really know about Electron, but I remember this talk about building a web browser in Electron. The security engineer actually recommends not try to do it. Their company tried it and it didn't go so well.

@nl255
Copy link

nl255 commented Jan 23, 2019

Option 1: Use the new API
If the new API is really weak, then I don't think Firefox can survive for long. If websites can more easily get around the weak APIs of Chrome, then website owners will have a motivation to push everyone to use Chrome. And soon Chromium-based browsers will probably become the only usable browsers.

Are you saying that websites will simply block Firefox users entirely if Firefox insists on keeping the existing webRequest API?

Option 3: Make a native app
There's a shell there's a way. I'm sad to see that there's no open source native app that does more than DNS blocking.
A native app is the best way to get around the constantly changing browser APIs, and as a bonus, it'll probably work for all browsers and even non-browser apps.
I'm not sure how hard this will be though, last time I tried it didn't go so well.

For this to work it would need to support HTTPS interception/rewriting which while it can be done opens up a huge can of worms security wise, especially if it is not done properly. It would also require having a companion GUI app for creating and installing the custom SSL certificate used for said interception as most people aren't going to be able to do it on their own using the standard command line tools. Even worse, on Android by default apps will not trust user installed root certificates (Chrome does but many other browsers and apps do not) and will need to be patched (unless the device is rooted, then the certificate can be installed to the system certificate store either directly or with a Magisk module).

@mikhoul
Copy link
Author

mikhoul commented Jan 23, 2019

@jspenguin2017 I have my answer for a fork on chromium to bypass a weak API if it was needed it'S possible and Brave seem to already use this strategy:

A developer on the Brave team was explaining it on Reddit:


_Bat-chriscat
BAT TEAM[M] 34 points 1 day ago* RemindMe!

It's worth noting that our Brave Shields (ad blocker) is not an extension; it is natively implemented. So extension API changes leave our shields unaffected.

Edit: We can always remove any code or update we don't like from the Chromium base we use. So even if this didn't just affect extensions but something deeper, we could just exclude it._


Source: https://old.reddit.com/r/brave_browser/comments/aijqm4/chrome_may_soon_change_how_3rd_party_ad_blockers/eeopqpb/

So maybe if they already do that you could fork Brave to integrate natively Nano inside and bypassing the API to intercept the calls you need ? 🤔

This way it would be future proof and the control would be very granular to block everything. User would just install the browser and use it. ☺️

Regards :octocat:

@nhantrn
Copy link

nhantrn commented Jan 23, 2019

@jspenguin2017 You might want to checkout Proxydomo hosted here on github. https://github.com/amate/Proxydomo

It's a local proxy with https filtering capability. I've been using it to strip out <script /> among other filters of blacklisted pages before it even reach the browser. It's a reverse engineered program of the popular Proximitron in the early 2000s.

There's also a recent separate effort called Proximitron Reborn, but I think the author of that hasn't release the source she promised yet. On going discussion of it is on the ancient prxbx forum here: https://prxbx.com/forums/showthread.php?tid=2331

@mikhoul
Copy link
Author

mikhoul commented Jan 24, 2019

Just to let know user that even the security guys at Mozilla seem to think it's a good Idea to weaken the API for extensions: https://old.reddit.com/r/firefox/comments/aithmh/raymond_hill_creator_of_ublock_origin_ubo_and/eerce78/

Firefox seem to be just an inferior "Chrome clone" right now with no real differentiation and no advantage sadly.

Keeping a strong and better API than Chromium could have been their Hail Mary but hey it's Mozilla as usual AKA stupid.

Regards :octocat:

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 24, 2019

@onmyouji
The security engineer actually recommends not try to do it.

Yes, I'm aware that a browser based on Electron is hard to secure, but so is extending a browser using native code.

As I stated before, I have no plan in pursuing this option. Thanks for the information though.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 24, 2019

@nl255
Are you saying that websites will simply block Firefox users entirely if Firefox insists on keeping the existing webRequest API?

That's a possibility.

For this to work it would need to support HTTPS interception/rewriting which while it can be done opens up a huge can of worms security wise, especially if it is not done properly.

I'm aware of it, that's why I want to use a well established project to handle the HTTPS repacking part.

If done right, HTTPS repacking can be stricter security wise, as you can choose to drop support for weak encryption before major browser vendors do.

Security is hard though, so we definitely need to be careful about this.

It would also require having a companion GUI app

We'll worry about that part later. That's going to be easy compared to the rest.

Even worse, on Android by default apps will not trust user installed root certificates

I thought that's only for apps that use certificate pinning. Either way, let's get it working for desktop Linux and Windows first, we'll worry about Android later.

@mikhoul
Copy link
Author

mikhoul commented Jan 24, 2019

Something that passed under the radar this autumn Mozilla want to follow V3 manifest too: https://blog.mozilla.org/addons/2018/10/26/firefox-chrome-and-the-future-of-trustworthy-extensions/

So the comment of Mozilla on Reddit is logical: https://old.reddit.com/r/firefox/comments/aithmh/raymond_hill_creator_of_ublock_origin_ubo_and/eerce78

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jan 24, 2019

@mikhoul
it'S possible and Brave seem to already use this strategy

I knew it's possible, and I knew Brave is doing it. That does not change the fact that coding in native code comes with risks, especially when the native code interfaces with untrusted inputs (web traffic).

@joey04 (from other thread)
But a browser fork is a massive effort.

It doesn't have to be. As you mentioned, the hard part is already taken care of by major tech companies and their paid developers. All we need is to insert our code at the right place and keep the fork up to date with upstream.

There are 2 problems though:

  • The code that we insert probably have to be native code, which can introduce new vulnerabilities
  • It's hard to get WideVine (DRM) working, also, Chromium depends on Google API a lot, you can get the API keys for your personal use fairly easily, but if you distribute your build the quota will run out pretty quickly

I won't say it's impossible, but it's hard enough that we want to explore other possibilities first. It's not necessarily a massive effort, but it'll probably be a massive paperwork.

@jspenguin2017
Copy link
Member

@nhantrn

Noted. Thanks for the information.

@mikhoul
Copy link
Author

mikhoul commented Jan 24, 2019

@jspenguin2017 commented on 23 janv. 2019 à 22:22 UTC−5:

@mikhoul

it'S possible and Brave seem to already use this strategy

I knew it's possible, and I knew Brave is doing it. That does not change the fact that coding in native code comes with risks, especially when the native code interfaces with untrusted inputs (web traffic).

@joey04 (from other thread)

But a browser fork is a massive effort.

It doesn't have to be. As you mentioned, the hard part is already taken care of by major tech companies and their paid developers. All we need is to insert our code at the right place and keep the fork up to date with upstream.

There are 2 problems though:

  • The code that we insert probably have to be native code, which can introduce new vulnerabilities
  • It's hard to get WideVine (DRM) working, also, Chromium depends on Google API a lot, you can get the API keys for your personal use fairly easily, but if you distribute your build the quota will run out pretty quickly

I won't say it's impossible, but it's hard enough that we want to explore other possibilities first. It's not necessarily a massive effort, but it'll probably be a massive paperwork.

I'm not sure on how hard it is for Widevine since https://chromium.woolyss.com/ have all the version build in almost all flavors quickly since many years with Widevine, if it was complicated they would not distribute it so widely.

But I understand that other API have quotas limit or a restricted but I never suffered from it but like I said I don't use the sync.

I browsed quickly https://chromium.woolyss.com/#google-api-keys and found it's relatively easy to install API key if you really need it, it seem it's a thing you do once and forget about it. 😏

I would really like a Chromium fork for power-users on desktop with a more compact GUI with less padding and above all with Nano integrated.

I know it's not the subject but take a look at CentBrowser they added lot of features for power-users. but he is closed source that's the downside but I use it with other forks.

Tomorrow I will ask to the CentBrowser team what is their take about V3 Manifest and restriction, they are a small team (~3 people) but they are responsive and open and I will report.

Regards :octocat:

@nl255
Copy link

nl255 commented Jan 24, 2019

@jspenguin2017

Even worse, on Android by default apps will not trust user installed root certificates

I thought that's only for apps that use certificate pinning. Either way, let's get it working for desktop Linux and Windows first, we'll worry about Android later.

Nope, it applies to Nougat and above. Even worse, there is no reason to think that Chromium and Firefox won't do something similar as they have already started heading that way with DNS over HTTPS (which also conveniently gives google access to all your DNS requests even if you run your own DNS server). So you might end up having to make either a fork or possibly a program loader (since DLL injection is blocked that leaves a loader program that patches the browser in memory) but of course with a loader they could fight back by using various tricks to make pattern based smartpatching fail possibly even including a Denuvo style custom virtual machine.

.

@elypter
Copy link

elypter commented Jan 24, 2019

even with all the harmful decisions firefox is still the better choice. i think it is more likely that firefox at least keeps the api somewhat better than chrome. and even if not, there are also firefox forks like for example https://github.com/intika/Librefox/issues which aims to create an always up to date version with an automated build system

@nl255
Copy link

nl255 commented Jan 24, 2019

Even with the potential issues mentioned earlier I would think the native, standalone app would be the best choice as it will be far easier to maintain a fork that merely makes SSL interception easier than it would be to reimplement the existing webRequest API especially if the browser code ends up being refactored to make it harder to do so as short of going proprietary or heavy code obfuscation/encryption it will always be easier to comment out a few functions regarding SSL certificate checks or make them always return "true" even if the check failed than it will be to keep the webRequest API working.

@mikhoul
Copy link
Author

mikhoul commented Jan 24, 2019

standalone app would be the best choice as it will be far easier to maintain a fork

I'm not sure about it at the end you will need an extension to be able to select element on the fly and create filters, even AdGuard that use a stand alone app need to have an extension and the dev already stated the such change in the API will hurt AdGuard.

@joey04
Copy link

joey04 commented Jan 24, 2019

@joey04 (from other thread)
But a browser fork is a massive effort.

It doesn't have to be. As you mentioned, the hard part is already taken care of by major tech companies and their paid developers. All we need is to insert our code at the right place and keep the fork up to date with upstream.

"All we need" eh? :-)

Kidding aside, I understand what you mean. Perhaps time-consuming is a more accurate term for a Chromium fork with the level of custom code we'd want for robust content filtering.

Just building a Chromium pull alone can take an hour or more depending on your CPU. And in the long run, there's the possibility of Google changing something internally that would impact an integrated custom filter engine. That could be a nightmare to deal with.

So I'm still of the opinion that modifying the Brave engine, which they generously share as MPL, is the best approach. It's already integrated into Chromium, and if Google breaks that interface in some way, the Brave devs will have to deal with it.

As for the other proposed idea, intercepting security certs sounds too much like SuperFish. I don't think I'd be comfortable with a benevolent FOSS one.

@nl255
Copy link

nl255 commented Jan 24, 2019

I'm not sure about it at the end you will need an extension to be able to select element on the fly and create filters, even AdGuard that use a stand alone app need to have an extension and the dev already stated the such change in the API will hurt AdGuard.

It could be done without an extension, it would just be uglier and depending on how it is done would likely require customization for each browser and major browser revision. Most browsers have an inspect element right click function which will bring up the source for the element in the developer tools which could then be OCRed via screen grab and compared to the page source captured by the standalone app. Or possibly by injecting custom javascript (AJAX?) code that communicates with a locally running server to get around the lack of sufficient APIs whenever the option to choose an element to block is selected. Or worst case the user would have to find the element using the inspect element and create the filter on their own (meaning only advanced users who know html would be able to block elements on their own). Though that could probably be worked around by using a fork that has dll injection detection/prevention disabled combined with an injected dll (or equivalent for other operating systems) meaning you would have to install Nanofox or Nanonium to be able to create your own element filters through a point and click interface.

@mikhoul
Copy link
Author

mikhoul commented Jan 24, 2019

It could be done without an extension, it would just be uglier and depending on how it is done would likely require customization for each browser and major browser revision. Most browsers have an inspect element right click function which will bring up the source for the element in the developer tools which could then be OCRed via screen grab and compared to the page source captured by the standalone app.

At his point it would be lot more effort and time that forking Chromium.... 😵

Also most users are not interested in opening Dev tool to hide an element, it's not convenient at all. I do that myself but only for really complex ads, 99% of the time I use the selector mode of Nano.

The goal of @jspenguin2017 seem to be to lower the load as much as humanly possible so i'm really not sure.

Also @joey04 have a really good argument it's really not everybody that would be willing to install a root certificate for an Adblocker I don't even trust AV to install a certificate.

Regards :octocat:

@imtbl
Copy link

imtbl commented Jun 20, 2019

I've made the switch to Brave on all my devices (macOS/iOS/Win 10) recently and while there's some things that are still inferior/missing in comparison to Chrome, for me personally, it made more sense than starting to use Firefox (where I was missing extensions and didn't like the dev tools as much).

Plus, I support Brave's content monetization ideas; they are not a perfect solution, but any alternatives to privacy-invasive ads/trackers are welcome.

So, I hope that you will continue developing Nano for Chromium-based browsers like Brave that will bypass the V3 changes, regardless of what happens with Chrome.

@mikhoul
Copy link
Author

mikhoul commented Jun 20, 2019

@mserajnik commented on 20 juin 2019 à 14:42 UTC−4:

So, I hope that you will continue developing Nano for Chromium-based browsers like Brave that will bypass the V3 changes, regardless of what happens with Chrome.

From @jspenguin2017 mouth:

Considering the code for the current webRequest is already public, there isn't a reasonable way to stop forks from using it.

Brave is not the only one who stated that they will keep alive webRequest API Vivaldi too: https://www.theinquirer.net/inquirer/news/3077032/vivaldi-promises-to-restore-third-party-ad-blocking-to-chromium

But we need more than webRequest API we need other API/methods that will be deprecated too, but Brave said that he will keep them too along with the webRequest API.

At the end of the summer I will switch to Brave or Opera too for my desktop use, worst come to worst it will be Firefox.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jun 21, 2019

we need other API/methods that will be deprecated too

Tampermonkey needs to execute remote code, which Google will no longer allow. Note that Mozilla never allowed executing remote code, but they made exceptions for those popular userscript hosts.

I'm not aware of any other APIs that we need being removed at this point.

@mikhoul
Copy link
Author

mikhoul commented Jun 21, 2019

@jspenguin2017 commented on 20 juin 2019 à 20:58 UTC−4:

we need other API/methods that will be deprecated too

Tampermonkey needs to execute remote code, which Google will no longer allow.
I'm not aware of any other APIs that we need being removed at this point.

The founder of Brave told that he will keep the permission to use remote code so extensions like Userstyles Managers & Userscripts managers will be able to exist, I'm pretty sure it will be the same for Vivaldi but I'm not 100% sure for Vivaldi.

The other things that many devs complaint is the removal of "background page" but I'm not sure if it will be replaced or not with others alternatives or not

I remember that Manifest V1 was completely removed from Chrome code only 12-18 months ago so we should have at least 12-24 months after the approval of V3 before they remove it completely from Chrome.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jun 21, 2019

many devs complaint is the removal of "background page"

Nano does not strictly need a background "page", it is a good-to-have, but not required. The new background service worker should be enough.

The only time Nano uses the background page renderer is to validate CSS selectors, but there are many alternatives:

  1. Borrow the renderer from one of the open pages
  2. Create a temporary tab / window when needed (easiest)
  3. Ship a library / polyfill (fastest)
  4. Use a web service

One crazy idea is to ship a small part of Chromium renderer as WebAssembly, and load / unload it as needed.

@nl255
Copy link

nl255 commented Jun 21, 2019

@jspenguin2017 I still think that the standalone app is probably the way to go especially if uMatrix (NanoMatrix?) and Tampermonkey/Violentmonkey like functionality can be added later as well.

After all, even without any special effort by google to complicate things it will be increasingly difficult for the various forks to maintain said compatibility as well as having to keep the extension updated on half a dozen to a dozen different addon/extension "web stores" assuming said fork even has their own.

@mikhoul
Copy link
Author

mikhoul commented Jun 21, 2019

@nl255 commented on 21 juin 2019 à 11:43 UTC−4:

@jspenguin2017 I still think that the standalone app is probably the way to go especially if uMatrix (NanoMatrix?) and Tampermonkey/Violentmonkey like functionality can be added later as well.

After all, even without any special effort by google to complicate things it will be increasingly difficult for the various forks to maintain said compatibility as well as having to keep the extension updated on half a dozen to a dozen different addon/extension "web stores" assuming said fork even has their own.

I don,t agree here, it would require a colossal sum of time and it would be really hard to maintain, All in one extension are the worst to maintain that's why UBO & Umatrix are 2 different extensions adding Tampermonkey/Stylus to the mix would be foolish IMO.

Moreover with a standalone approach you will lose lot of granularity and ability.

There is no real problem since 2 major forks (at least) are committed to keep the old API and extensions are far more "easy" to keep updated compared to a Frankenstein All-in-one standalone app 😉

@nl255
Copy link

nl255 commented Jun 21, 2019

@mikhoul Not compared to maintaining multiple standalone proxies for each extension and trying to get them to play nice when daisy chaining them together. As for those forks that is nice in the short term but it will become increasingly difficult to maintain those old APIs as time goes on. Not to mention that with Brave and other forks besides Opera how exactly are you going to get the extension since you won't be able to get it from google's "web store" anymore (unless they disable extension signing completely like Kiwi does)?

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jun 22, 2019

and it would be really hard to maintain

Depends on how the proxy engine is designed, it can be designed to be extensible. It can even be designed to emulate the current webRequest APIs.

Moreover with a standalone approach you will lose lot of granularity and ability.

I don't believe a MITM proxy has less granularity.
In terms of ability, blocking pop ups might be a bit hard. Nothing showstopping though.

@jspenguin2017
Copy link
Member

it will be increasingly difficult for the various forks to maintain said compatibility

If you're talking about legacy manifest v2 extensions, then yes. But for manifest v3 extensions that use blocking webRequest API, then probably not, since Chrome will still allow enterprise installations to use it.

having to keep the extension updated on half a dozen to a dozen different addon/extension "web stores"

Yes, that is certainly a problem. Especially when some stores don't support publishing from command line.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jun 22, 2019

I know it's been a while, but I have yet to make up my mind. I have been observing the situation develop though, and here's my update on the "3 options" I listed at the very beginning.

  • The new API is still underpowered, but becomes more and more viable with each update. I'll investigate deeper after another few updates to evaluate its viability.

  • As for forking Chromium, this is no longer needed. Instead, this option is replaced by moving to a browser that supports blocking webRequest in normal installation. This comes with its own issues though, like how to quickly publish updates to many extension stores.

  • I have explored making an native app, and I made an initial prototype which confirmed the theory. However, making a fully functional native app is a non-trivial challenge, and I'm not sure if I'll have enough time to finish it in time.

Here are the important things to watch next.

  • Microsoft's move. As the new Edge is getting closer and closer to stable release, Microsoft's response to manifest v3 is probably going to be the most impactful decision on this whole situation.

  • Upstream's move. Depending on what upstream does once Google makes the final decision, we may need to adjust our plan to minimize headache during the transition.

@mikhoul
Copy link
Author

mikhoul commented Jun 22, 2019

@nl255 commented on 21 juin 2019 à 13:29 UTC−4:

@mikhoul Not compared to maintaining multiple standalone proxies for each extension and trying to get them to play nice when daisy chaining them together. As for those forks that is nice in the short term but it will become increasingly difficult to maintain those old APIs as time goes on. Not to mention that with Brave and other forks besides Opera how exactly are you going to get the extension since you won't be able to get it from google's "web store" anymore (unless they disable extension signing completely like Kiwi does)?

Not really since Brave already maintain its own code for their own addblocking and for the webrequest API like @jspenguin2017 already stated the code will be in Chrome/Chromium and even it it was removed it's easy for Brave/Opera to maintain different API they both have already of custom API/mechanism.

how exactly are you going to get the extension since you won't be able to get it from google's "web store" anymore (unless they disable extension signing completely like Kiwi does)?

There is many way for me I just side-load in dev mode for the casual user Opera have already it's own store and it will probably be the same for Brave. With Microsoft browser it's the same they have thir own store for extension and you can choose from the Chrome store of the Microsoft store.

Brave is already ready to have their own store for the extension that would like to use their specific API/features: https://github.com/brave-experiments/brave-extension-store

Also there is nothing complicated to keep an extension updated for the user the update is automatic and for the Dev I'm pretty sure Brave and Opera will make the publishing process easy for the uploading/updating for the extensions that will take advantage of their particularity.

Don't forget that Firefox could also be an alternative again for power users IF they don't follow the path of Google for V3 API... However I will not hold my breath for Mozilla doing the "good" thing... I have been deceived to many times by their decisions.

BTW: Chrome team today stated again they want global limit for rules not per extensions which will limit a lot many extensions.

@nl255
Copy link

nl255 commented Jun 22, 2019

@jspenguin2017 As I understand it, the blocking webRequest API is only available in the paid Enterprise version and thus the code for it won't be present in the Chromium source code (i.e. the patch to enable/implement it will be closed source). Otherwise it would probably be fairly easy to patch Chrome or write a Chrome loader to enable it for all users*. Of course said loader would also need to disable signature checks or at least enable installing extensions permanently whenever in dev mode.

As for the native app, if you do decide to go that way you might want to contact the EFF as their privacy badger addon has the same issues so perhaps they can help finish it so they can use the same code as well.

*While Chrome does have some dll injection protection it probably doesn't have anything even on the level of Shiva to prevent in memory or even on disk patching much less Denuvo level. You could probably even use smart patching (i.e. pattern based rather than offset based patching) to make it work with multiple versions.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jun 22, 2019

it's easy for Brave/Opera to maintain different API

If they roll their own APIs, I would count that as a fail. That would quickly become too fragmented.

I'm pretty sure Brave and Opera will make the publishing process easy

I only heard about complains about the Opera store. Don't keep your hope up.

Chrome team today stated again they want global limit for rules not per extensions which will limit a lot many extensions.

Since the per-extension limit would be lifted, I'm not sure if this is positive or negative. Note that the rules limit for Safari is 50k, and some people / newspapers consider Safari to be an adblocker-friendly browser.

@jspenguin2017
Copy link
Member

the blocking webRequest API is only available in the paid Enterprise version

While it is possible for Google to strip the code from Chromium and only ship it in Chrome, I think you can get the enterprise version for free.

Even if they do strip the code, I don't think they'll do anything to make it painful to re-add it as they would be stepping on their own toes.

@mikhoul
Copy link
Author

mikhoul commented Jun 22, 2019

One more desktop browser will officially support the old webRequest API: https://www.zdnet.com/article/opera-brave-vivaldi-to-ignore-chromes-anti-ad-blocker-changes-despite-shared-codebase/ 😄

@jspenguin2017
Copy link
Member

jspenguin2017 commented Jun 30, 2019

RegExp filters are being implemented:
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-extensions/0dKgL8sMbhg
https://docs.google.com/document/d/1foL79VwByaVeUwYPBfZeqyC2SYcAawhjjH05eqHqW_E/edit

@mikhoul
Copy link
Author

mikhoul commented Oct 12, 2019

Hi @jspenguin2017 ,

Not precisely related to Manifest V3 but it's in the same vein with Google attempt to restrict ad-blockers ...

From this Issue in UBO https://old.reddit.com/r/uBlockOrigin/comments/dgncga/dev_build_1225rc1_rejected_from_chrome_web_store/ I understand that UBO will possibly be no longer available from the CWS and I fear the same thing will happen to Nano.

What is your take on this @jspenguin2017 ? 🤔

Regards !

@jspenguin2017
Copy link
Member

@mikhoul Sometimes false positive happen, it usually gets resolved after a resubmission or an email. I don't think it's a problem.

@okiehsch
Copy link

usually gets resolved after a resubmission or an email.

uBlockOrigin/uBlock-issues#745 (comment)

@mikhoul
Copy link
Author

mikhoul commented Oct 12, 2019

uBlockOrigin/uBlock-issues#745 (comment)

Gorhill have a rigid personality and did not change anything in the code when he resubmitted so it's expected to receive the same results. Just changing some code "cosmetically" could have resulted in the the extension to have passed instead of being rejected.

Gorhill has many quality (very good developer, focused, hard worker) but he is not flexible and he is a little bit in his bubble, he did not even try to contact Google by other mean than answering to the email, he just marked the issue as "wontfix".

Anyway right now Google is looking in the issue: https://old.reddit.com/r/chrome/comments/dgoymg/warning_ubo_ublock_origin_will_possibly_be/f3e4fi6/

image

@okiehsch
Copy link

okiehsch commented Oct 12, 2019

1.22.5rc2 has been approved in the CWS.

did not change anything in the code when he resubmitted

After they rejected 1.22.5rc1 he submitted 1.22.5rc2 which was also rejected.
Here are the code changes gorhill/uBlock@1.22.5rc1...1.22.5rc2.

@mikhoul
Copy link
Author

mikhoul commented Oct 12, 2019

1.22.5rc2 has been approved in the CWS.

Good news so it was just some kind of mistake of a false positive like @jspenguin2017 was saying.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Oct 12, 2019

It's likely caught in some automatic filter, usually (so far, every time for me) sending an email to request a human review corrects the problem.

@okiehsch
Copy link

okiehsch commented Oct 13, 2019

Good news so it was just some kind of mistake of a false positive

I never thought it was some nefarious scheme to get rid of uBO but I find it puzzling that after nearly ten years of CWS a company like Google still has a system in place that makes it very difficult for developers to resolve issues with the automatic filter.
See the links at uBlockOrigin/uBlock-issues#745 (comment) for more examples.

@jspenguin2017
Copy link
Member

I feel that WebStore is easier to use (for developers) than Microsoft Store and AMO. Although it looks ugly, it is blazing fast. Microsoft Store is definitely the worst in term of performance, it takes forever to load and does not offer a functioning automation API.

As for clearing false positives, WebStore have been the easiest among the three in my experience, simply replying the automatic email is enough to get a human review. Unlike the other two which require submitting a new version with an explanation letter.

@okiehsch
Copy link

Point taken, you are clearly in a better position to make an argument about the quality of CWS than I am.
I can only go by anecdotal evidence and reading through dessant/search-by-image#57 especially the replies by google felt like a kafkaesque experience to me.

@jspenguin2017
Copy link
Member

jspenguin2017 commented Oct 13, 2019

Every store have some weird policies. Microsoft Store doesn't allow screenshots to have description on the top 75% of the image, but it's perfectly fine on the bottom. AMO doesn't allow $ to refer to a library that's not a trusted (hash whitelisted) version of jQuery, but it's perfectly fine after renaming the namespace.

I do agree that WebStore emails have poorer quality than the other two stores' and I do wish WebStore emails would contains proper description and actionable directives like those from the other two stores, but I don't think this is something to freak out about.

@jspenguin2017 jspenguin2017 changed the title Chrome Manifest V3 discussion mega thread Chrome Manifest V3 discussion megathread Apr 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests