Port to Firefox

[gorhill]

Never wrote an extension for Firefox, so that will require some reading (started). This might take a while.

[gorhill]

When v1.0 "ships", I will start working on this one.

This extension gives back so much control to the user, that I would like a much wider base of users as possible, so as to make it difficult for browser makers to take away the level of control users will take for granted with HTTPSB.

This is what worries me at this point: a change in API that would reduce the level of control a user has over its browser.

[johnny]

Thank you for this Plugin. I have yet to understand all its features, but having one plugin instead of several (NoScript + Adblock + Ghostery) to get sane web pages is a huge step forward.

Since I am using Firefox as my default browser, I would very much like to see this port.

[ghost]

While porting HTTPSB to Firefox is surely an interesting idea, I think you should be prepared that you will probably need to add features which are presently not available in HTTPSB.

The most important one is adding an XSS filter: Chrome/Chromium has the XSS Auditor while Firefox doesn't have such a filter. It's planned (https://wiki.mozilla.org/Security/Features/XSS_Filter) and on their security roadmap (https://wiki.mozilla.org/SecurityEngineering/Roadmap). Right now it's not available so you would probably have to add this feature (and presumbly other ones, e.g., something against clickjacking) in order to offer a comparable level of security as Noscript does.

I'm not saying this to discourage you from porting your extension. But there may be more work involved as it looks at first glance.

[Naganova]

Absolutely love the addon and cannot wait for the port but I kinda have to agree with tlu. But if this carried the few little safety features that NoScript has, it will be without a doubt, the best fine-grained security, request control addon out there. Perhaps you could integrate it, given that it's also open-sourced.

Godspeed Gorhill.

[idlewan]

To get started with the firefox extension, here's some link that might help the coureagous that will start porting it:

• installing the sdk: https://developer.mozilla.org/en-US/Add-ons/SDK/Tutorials/Installation
• writing your first addon: https://developer.mozilla.org/en-US/Add-ons/SDK/Tutorials/Getting_started
• intercepting page loads: https://developer.mozilla.org/en-US/Add-ons/Overlay_Extensions/XUL_School/Intercepting_Page_Loads

All filtering extensions (ABP, AdblockEdge, Ghostery, Disconnect, RequestPolicy, PrivacyBadger) use nsIContentPolicy and sometimes HTTP Observers with http-on-modify-request or http-on-examine-response. You can start by reading their source. The most readable is probably PrivacyBadger (https://github.com/EFForg/privacybadgerfirefox) because they seem to write lots of useful comments, plus they do also cookie looking/blocking and are the most recent (have less bagage and cruft) and are obviously actively maintained.

[xdmx]

👍

I'm impressed how fast you develop and how much the plugin improved since the last time I tried it (~2 months ago), it's now the best plugin I ever tried. As a Firefox user I would love to see it ported to replace my current noscript+adblock.

[gorhill]

Will start to address this soon (not the expected way though).

[gorhill]

What if:

Most of HTTP Switchboard became a local proxy?

• All browsers could benefit from it -- or actually whatever devices connecting to that local proxy.
• Performance because running using compiled code, meaning more freedom to implement more complex filter engines.
• UI would be served through a special port on the proxy.
• Communication with browser using special header entries (these would be stripped of course before requests send to external world), so browser can tell proxy how to sort out requests (which tab a request belong, etc.)
• Javascript can be disabled just the same way as of now, and works for all browsers because not relying on specific browser API (CSP is standard).
• More freedom for user various front-end, the proxy could serve many flavor of UI.
• Can report all requests, including those not routed through browser API -- the requests behind behind-the scene scope.

So mainly the browser-specific extension code would be reduced to a minimum, merely XHRing with the local proxy. For browsers with even not enough a good API to support that minimal extension code, there is always the fall back of using just a tab in the browser (instead of the extension) to interact with the HTTPSB-proxy.

Disadvantage is user would have to install the HTTPSB-proxy, not as seamless as installing an extension. This is something which bothers me because I want everybody to have the ability to be informed about what their browser does, not just advanced users. But could be mitigated by having well though installation -- I envision a single exec, no bundle, which would work out of the box, as feasible.

Foolish idea?

[jonvuri]

Main disadvantages I see personally at first glance:

1. A separate app, especially one that acts as an internet proxy, is a lot more complex than an extension. You already noted this, but it's more than just installation, Windows and OS X each have their own little quirks dealing with proxies, and it's another app to run while you browse the internet. And god help you if you need to make it work with another proxy or VPN.
2. Since it's not integrated with the browser you can no longer see just the filtering scope relevant to the single tab you're looking at. Unless the extension somehow sent the proxy sentinel requests every time you activated one, I guess. What if you have multiple browsers open? Now Switchboard needs to know which one is in the foreground too. Could probably work but would need a lot of magic.

[gorhill]

an internet proxy, is a lot more complex than an extension

I was looking at goproxy, so I wouldn't have to deal with that part from scratch.

Since it's not integrated with the browser you can no longer see just the filtering scope relevant to the single tab you're looking at. Unless the extension somehow sent the proxy sentinel requests every time you activated one, I guess. What if you have multiple browsers open? Now Switchboard needs to know which one is in the foreground too. Could probably work but would need a lot of magic.

Each request has a tab id, which is unique among all opened tabs (across all opened windows), so not an issue, just a matter of sending that information in a header dedicated for that sole purpose for each request, and given that all filters and scope data would be inside the proxy, there is not really any issue there. A companion extension would be merely a lightweight front end UI on the proxy, and served by the proxy (so matrix built by the proxy instead of the js code).

I wonder about VPN etc, but given that Go has a full HTTP server (on which goproxy is based), I am assuming the proxy sends the requests to OS, which would then deal with them from this point, so if there is a VPN or whatever, this will be all taken care.

[mnem]

Privoxy takes this approach and has been around for quite a long time, so it isn't that foolish an idea :) I think GlimmerBlocker on Mac is proxy based too in order to get around limitations with Safari and that's a fairly painless install. It's probably worth having a poke at those projects issues pages to see what sort of workarounds they need to use.

Proxy configuration varies a lot per-OS, but generally you can set the proxy in the browser somewhere, so not everything in the OS has to use it. For people using it at work, you may have issues with chaining goproxy to whatever their work proxy is, but I'm not sure how common work proxies are these days. They're probably only used in big companies , and they likely restrict installing things locally anyway so probably don't matter :)

[ghost]

an internet proxy, is a lot more complex than an extension

I'm very open to this idea of either the stand alone library on #332 or the proxy idea just like Privoxy and I would say it's very welcome indeed since there hasn't been an update on the project since there hasn't been any major activity on privoxy since 2013-10-18 I would say it's worth at least a try.

[ghost]

The problem with a proxy like Privoxy is that it would have to be available for different operating systems. I'm not sure if Raymond wants to do this ...

[gorhill]

Go code is portable out of the box (unless of course you use OS-specific stuff). It's actually simpler than having to support various browser extension APIs (which may also change). On top of that this comes with so many benefits, I do not see myself porting code to specific browsers, except for a "proxy extension" (if I may say) to interface the proxy and to provide it with useful data to better categorize net requests. Then there are other benefits: all browsers supported, all behind-the-scene requests reported and filtered if user choose to, speed and hence more flexible filtering engine, inline js can be blocked regardless of browser API limitations, etc etc

[jonvuri]

@gorhill Sorry, can you clarify if the new core filtering engine is to be pure JS or if you meant that it would be this new external proxy app?

[gorhill]

new core filtering engine

We are mixing things up here. The issue here is to find a way to have HTTPSB available to Firefox in particular, and other browsers in general.

Issue #332, "Spin-off a standalone filtering library", is not about giving HTTPSB a "new core filtering engine", it's just to make available to other developers an optimized ABP-compatible filtering engine. I haven't seen the various ABP-lookalike extensions out there use the various optimization solutions in that library, so it would no doubt benefit the users to see it used in other blockers. If anything, performance for many of them are worst than ABP itself (Adblock is abysmal). My motivation is, 1) as a developer I want my stuff to be used, 2) there is definitely a need for efficient ABP-compatible filtering so as to shake up the current statu quo of inefficient solutions as if nothing more can be done about it.

[fijam]

A modern reimplementation of Privoxy would be certainly welcome by many technically-inclined users, however, if you are concerned about adoption, the barrier of entry for regular users will be higher.

In addition, many people desire adblocking in corporate environments where they might already be forced to use a proxy. Chaining proxies for a regular Joe would be a challenge. And if something breaks, you can't just remove a browser add-on. The installation of standalone applications (esp. requiring administrator privileges or network access) on company computers may also be prohibited.

On the other hand, how many browser APIs do you really need to support? Chrome+Firefox puts you already at over 60% of the market.

[idlewan]

the barrier of entry for regular users will be higher. 

Regular users don't use Adblockers. Who gives a damn that people that have no chance on stumbling on the proxy/extension and don't care about privacy can't download an executable?

 The installation of standalone applications (esp. requiring administrator privileges or network access) on company computers may also be prohibited. 

Are these people even able to install a browser extension? I suspect their 'company computers' usually run older browser versions that don't auto-update. Again, who cares?

Having HTTPSB as a proxy (that can itself be configured to use another proxy) allows much more than an extension: as already said earlier in the thread, you could configure your tablet/phone to use the proxy. These usually don't have ways of installing extensions (correct me if I'm wrong) or even the processing power required to run them.

Caring for "the regular user" is not interesting in my opinion: they can always use Adblock Edge or PrivacyBadger. This is about making something better (performance, abilities).

I'm all for a modern, performant Privoxy with the inspection power of HTTPSB.

[fijam]

Regular users don't use Adblockers.

85% percent of Firefox users have at least one add-on. The most popular one is Adblock Plus. Take it for what you will.

Who gives a damn that people [...]

I assume the developer himself, so don't pick the bone with me. Two posts up:

as a developer I want my stuff to be used

There is a reason why Privoxy is a footnote in history and why gorhill was not even aware of it, and it's not some grave technical inferiority.

If he wants to support the maximal number of users, there is also the need to support those in corporate settings or stuck on IE (20%). In a roundabout way, I am making the point that those souls are lost already as they wouldn't be able to run a standalone application anyway, so the 60% of Chrome+Firefox might be "good enough". However, it's neither mine nor your call to make. I just put it in perspective.

[idlewan]

Alright, I was using a different definition for "regular users".

What I meant was that you can't win the popularity contest against Adblock Plus for "regular users that want an adblocker" ("can't configure their proxy settings" type) with something that is just about the same performance-wise (I'm talking about Firefox and other browsers here. As demonstrated in the benchmarks, HTTPSB has no problem being more performant than Adblock-* on Chromium). It's just not possible to dethrone the market leader just like that.

You need to be inherently better, and that is achieved the best through an external application that takes all the performance it can get from the hardware (a compiled program). You might still need a noob mode for the interface (click install and forget about it), because that's what "regular users that want an adblocker but don't want to be bothered about details" probably expect.

I still think not caring for users that can't use an external program is not a big deal, because they can use something else (AdblockPlus, Adblock Edge, Privacy Badger), and that's fine.

[gorhill]

Note that I experimented with goproxy, and I did not have to perform any administrative steps, it works under user privilege -- that's why I call it lightweight. Go library contains a standalone HTTP server. This also means no special configuration (my understanding so far), goproxy will itself use whatever is the current outgoing network connection. The only special step needed was to tell the browser to trust the custom certificate used by goproxy in order to be able to work properly with encrypted connection.

I did install Privoxy, and first I was presented with instruction telling me I needed to sudo some system config file. That won't work. Working with goproxy as a basis completely eliminate this step.

[equim]

I'm very happy to see you consider a proxy gorhill.

For a while now I've been thinking that I really want most browser functionality in a proxy. It's always seemed strange to have bookmarks, history, tabs, cookies, saved form data and extensions like greasemonkey, stylish and http switchboard be something that is part of the browser. The web is something we typically access from many devices and you want to access the same content and functionality from all of them. Some have tried to solve this with browser sync and/or the cloud which generally don't do everything, are very proprietary and out of the user's control.

I am very much hoping for a world where we all have our own proxies which can sync and share content with each other

[jonvuri]

history, cookies, form data

I don't know about you but I'm not trusting that data to any application that requests it.

[equim]

I don't know about you but I'm not trusting that data to any application that requests it.

I'm not sure what you mean. What application would request it? I want the proxy to take on that part of the browsers functionality - it wouldn't request the data from the browser, it would simply have the functionality to store and manage the history, cookies, form data etc itself.

I think I'm dragging this off topic though - I really just wanted to say that I think a proxy implementation of http switchboard is a good idea because it fits in with my own thoughts on where most browser functionality should be.

[somini]

The thing is, without the matrix, the proxy is just a glorified AdBlock Rules parser. Not sure if it's worth all the work, Firefox users already have the AdBlock proper.

[gorhill]

@somini: I never said the matrix would not be implemented by the proxy. There can be a specific URL on the proxy to access a specific matrix (as per URL parameters) which will be served as a HTML document (just like the current matrix is, a HTML document). An extension would query the matrix from the proxy with proper URL. That's actually the part I like, it's also likely to be generated faster using native code than js code.

[ghost]

An extension would query the matrix from the proxy with proper URL. That's
actually the part I like, it's also likely to be generated faster using
native code than js code.

Sounds good! But what about applications (like email programs or RSS
readers) for which no extension will be available? I guess that they
will benefit from this proxy, too, as the proxy would still block all
those domains in the ubiquitous lists, won't they?

[gorhill]

I guess that they will benefit from this proxy, too, as the proxy would still block all those domains in the ubiquitous lists, won't they?

If they are set to go through the proxy, yes. Remember, for me a lightweight proxy means not a system-wide one, so you have to tell each app to go through it if you want so. benefit is that it doesn't interfere with system wide internet settings (vpn, etc.)

[somini]

@gorhill I misunderstood then, I apologize. This gets better and better.
As for the lightweight proxy I agree, since there might be incompatibilities with specific programs.

[Zzyzx-]

Has there been any new progress on this?

I think it would be a bad idea to do a proxy. It would make it difficult to do on sites with ssl and impossible to do with sites that are pinned to specific certificates. https://code.google.com/p/chromium/codesearch#chromium/src/net/http/transport_security_state_static.json&l=183

[gorhill]

difficult to do on sites with ssl

Goproxy worked all fine with SSL after importing the built-in certificate in the browser.

[kuntau]

Please go for it @gorhill. As long as it is cross platform I'll reserve the critic after I use it.

[my-password-is-password]

I saw this firefox extension called Policeman that kinda looked like it was trying to do what httpsb is doing. Never used it though. Doesn't look very user friendly.

https://addons.mozilla.org/en-US/firefox/addon/policeman/

[naturallymitchell]

I agree that externalizing httpswitchboard's functionality would be wise.

Please also consider one more implementation by reviewing http://www.slideshare.net/TonyFabeen/devinsampa-nginxscripting & https://ef.gy/using-nginx-as-a-proxy-server

I believe these show Nginx's power beyond its most common use cases (when combined with Lua scripting) and advance a highly beneficial approach.

There is only one limitation I see, which is that SSL support, via CONNECT method, is currently unsupported in Nginx, so perhaps that feature could come by way of goproxy (in a proxy chain), where needed.

[somini]

@my-password-is-password I use policeman and it's like HTTPSB without the AdBlock part. Reading the AdBlock rules is planned.
I think it's about as user-unfriendly as HTTPSB. It's a tool for power users.