Reference answer to "Is it possible to automate 'purge all caches' + 'update now' [every x minutes]?"

[gorhill]

Just to have a reference answer to this commonly and trivially suggested "feature":

That's why not:

This is the cost of how much bandwidth uBO currently incurs on each server freely hosting the filter lists -- keep in mind these lists are updated every few days. The lists are currently freely hosted on a couple of servers, each of those servers see the kind of bandwidth shown above if not more. And notice that the price tag above is only for North America.

If we let this sort of bandwidth usage grow unbound, it's not going to be free anymore, and you can see the sort of cost we would face -- it's essentially an end to the project.

Version 1.54.0 of uBO will support differential updates, this should significantly reduce bandwidth while having lists update more often.

[gwarser]

https://www.cdnperf.com/tools/cdn-calculator

[iam-py-test]

FYI the CDN statistics image is broken

Thanks

[gwarser]

Probably because it was originally posted in private discussion https://github.com/uBlockOrigin/uBlock-discussions/discussions/781#discussioncomment-7414674

[gwarser]

It was based on jsDelivr data from last month https://www.jsdelivr.com/package/gh/uBlockOrigin/uAssetsCDN?tab=stats

And Google Cloud calculator was used https://cloud.google.com/products/calculator#id=3df4242f-5e4b-4b2d-96e7-d87a0c2a1a8b

[foxjaw]

I always thought default filters on uBO are retrieved through the source repository (uAssets). That would meant I thought github lifted all the filter extraction work. Isn't that true ?

[gorhill]

They are fetched from one of many servers mirroring GitHub repo: https://github.com/gorhill/uBlock/blob/1.53.0/assets/assets.json#L49-L54. At the bottom of About pane, it says:

uBO's own filter lists are freely hosted on the following CDNs:

Cloudflare Pages, GitHub Pages, jsDelivr, Statically
A randomly picked CDN is used when a filter list needs to be updated.

[megamorphg]

How about having a sign-in mechanism in order to update the lists? (and a throttle per registration)
Otherwise forcing user to manually download the resources and upload from the site without a sign-in.
It would be interesting who/what group is consuming most of this bandwidth.

[gwarser]

This will be against policy to not collect any data.

[GunGunGun]

How about implementing IPFS Update for future ? It will allow users to share bandwidth instead of using the same CDN again and again, our bandwidth is unlimited because we have a bunch.

[Kishlay-notabot]

Same thought... But the main question comes is that we would have to restrict any form of tampering done to the filter lists, let's say if a person edits a filter list and removes some links, and it gets shared?
You still need a parent server to keep a check (cross-check) on the version or any directly related parameter of the filter list' content
It might be a dumb doubt but yes IFPS might still need some upper level auth or comparison standard

[dido04031983]

Perhaps secure hash algorithms can be put to play just like ones in p2p.

[krystian3w]

Github page (https://ublockorigin.github.io/uAssets/update-lists.html) are protected before injection in loop own trusted-click-element?

https://stackoverflow.com/questions/66980994/simulate-trusted-user-click-in-javascript

[gwarser]

Script is watching for isTrusted event. https://github.com/gorhill/uBlock/blob/00fb5f18b2d2d9f91bcefb7adfda8e16b8369132/src/js/scriptlets/subscriber.js#L81

[krystian3w]

So only fail with macro for mouse? Like Auto Hot Key and Auto IT?

https://www.autohotkey.com/
https://www.autoitscript.com/site/

[krystian3w]

document.addEventListener('click', ev => {

click cover spacebar / enter? If nope then maybe better check oinput? https://www.javatpoint.com/oninput-function-in-javascript:

document.addEventListener('oinput', ev => {
// ... TBC ...

[oSumAtrIX]

Cloudflare and co. provide caching mechanisms for free before requests hit your server suitable for hosting the lists. With proper cache rules the bandwidth can be reduced by 99% if not more. Under my impression I must be missing something?

[GunGunGun]

Yes, you missed one important point:

• Cloudflare is privacy invasive
• Cloudflare blocks IP randomly
• Cloudflare blocks non-human traffic, bot-like traffic

So dealing with Cloudflare isn't worth it, I rather implement IPFS and let everyone contribute their own bandwidth instead.

[gwarser]

https://ublockorigin.pages.dev/ backend is Cloudflare.

---

AFAIK their "free" service disallows hosting files, that's why there is a landing page in root, but we don't know if it's enough for them, and they can shut it down at any time.

[krystian3w]

ELA is twice on unsorted list, really needed is dupe without click option?

[alexandreteles]

https://ublockorigin.pages.dev/ backend is Cloudflare.

---

AFAIK their "free" service disallows hosting files, that's why there is a landing page in root, but we don't know if it's enough for them, and they can shut it down at any time.

You are correct about that. They do, however, allow for almost unlimited caching even at the free tier. I think that @oSumAtrIX question was related to the possibility of slapping CF in front of a given endpoint and have it serve most data from their edge cache, what would massively reduce the bw requirements on the servers.

Their API also allows for cache to be invalidated on demand whenever there is a change to the underlying data so cache inconsistency isn't too bad to manage either.

[HappMacDonald]

What is involved for someone to become a CDN partner to help share the load on bandwidth for things like uBO filter lists or other aspects of your operation?
Like, are there minimum BW speed, volume per time, storage at rest requirements? Must certain kinds of software be run on the CDN servers to best support your files being mirrored to and served from them?

[GunGunGun]

Usually it's best to not involve any partner unless data hosting cost increases significantly, because using 3rd party hosting is risky, just take a look at xz scandal happened a few days ago, it was super dangerous, the whole internet could've been pwned if someone didn't find out the backdoor, because it's so sophiscated and it could infect all Linux system with sshd turned on, and take over most of the internet.

[HappMacDonald]

It doesn't sound to me like it has to be that dangerous, depending on what the download client is and how that operates.

I'd imagine that if you do not want to have to trust the storage source (AS WELL AS the transit pipe: people log into wifi hotspots at coffee shops after all, don't they?) then the download client merely has to be initially retrieved at a trusted source and include a signing key. And if you're able to trust the initial download source well enough you wouldn't even need a CA-certified signing key for that to work.

Then any time you're pulling an update, you just verify the downloaded data via that signing key before acting upon it.

Regardless, it sounds as though the data is already being stored on third party CDNs so I'm still just asking what the standards they are being held to already is. Obviously the ones already in use are trusted, so how do they prove their trustworthiness?

My perspective is that I work at an ISP that pays a lot of money to have lots of Gbps of downstream service for our users, but that can only be purchased along with an equal number of Gbps of upstream availability that our users leave 90% unused (not least of which due to last-mile constraints). So every time I see figures like the top of this thread that say "$6k for 100GB of upstream data transfer in a month" it reminds me how we are presently capable of delivering a thousand times that per month but lack the rubber-meets-the-road knowledge of how to actualize it. Be it for direct profit, or even for volunteering such a resource to a non-profit cause.

So I want to find out as much as possible about the landscape that might lie between where we are now and being able to get some use out of our wasted upstream potential.

[stephenhawk8054]

Can you show real examples of how your CDN is running for other repos? And how can a person register to use it for free? Currently, besides Cloudflare that needs a free account (gorhill registered for it), uBO is using jsdelivr and statically.io who allow any github repo to use it.

What uBO is finding is a CDN that allows to transfer ~100TB bandwidth and ~1.5B requests per month for free.

(Just a little nitpick: the image in the 1st post is not "100GB a month", it's about 100TB a month).

[HappMacDonald]

Ah ha, my bad then. I was misreading the comma in the image as a period. 😅
That does put it out of the range we'd be in a position to directly assist with after all. Thanks for setting me straight though.

[ryanbr]

Just to outline,

Brave just pull from Github directly, not from the uBO mirrors. https://github.com/brave/adblock-resources/blob/master/filter_lists/list_catalog.json, just to clear the air. Also Brave will download the lists, and users will grab from Brave itself and not the source urls.

Does jsdelivr allow config of traffic, see any potentional abusive IP details. and also rate limit abusive ip's/users? Or ask if its possible if the same (possible abusive) IP requests it within X mins/hours, taking into account companies with static IP's etc.

Then it can be slowed down or blocked until the requests stop coming through after a certain time frame. Maybe reachout to see if jsdelivr can help? and point to this thread.