Find a better way to block google play test device #229

riderx · 2023-05-11T15:28:32Z

For a period i did set a ip ban for google IP sadly it was also blocking some user with vpn.
i need to find a better option

riderx · 2023-08-09T14:55:54Z

/bounty $100

algora-pbc · 2023-08-09T14:55:56Z

💎 $100 bounty • Capgo

Steps to solve:

Start working: Comment /attempt #229 with your implementation plan
Submit work: Create a pull request including /claim #229 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Additional notes:

ℹ️ If something is not clear ask before working on it, otherwise your chance to rework it is high
🎥 To claim you need to provide in your PR a demo video of the change
👨‍👩‍👧‍👦 Join the Discord to get help
📏 Check all Bounty rules

Thank you for contributing to Cap-go/capgo!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🔴 @rishi-raj-jain	Aug 13, 2023, 6:21:48 PM	WIP
🔴 @ologbonowiwi	Oct 13, 2023, 7:19:17 PM	WIP
🔴 @WcaleNieWolny	Oct 13, 2023, 7:24:33 PM	WIP

rishi-raj-jain · 2023-08-13T18:21:46Z

/attempt #229

where do we want the ban to be applied? on the web? more details would help!

Options

Cancel my attempt

riderx · 2023-08-14T11:43:22Z

this is for the update endpoint.
when users have CI/CD to publish to google play store then google play is starting the app and request live update. who are counted in users billing plan.
So the goal is to be able to know what is a google bot device and block the update, to not count in billing

rishi-raj-jain · 2023-08-14T12:07:40Z

On it!

WcaleNieWolny · 2023-09-02T18:16:21Z

This is partially fixed by #314 correct?

if redis does get merged can I just add a fn in the updates_cached that will call supabase, check if device exists in cache and if it does not then return the cached version (if possible). Here is how I see it:

1. Only if app exists in redis and the requested version is cached
2. Check if device exists in cache, if it does not create it as `standard` (not overwritten) and set some sort of value that indicates the requests (like for example `device_{ID}_usage` to 1
3. When this usage > 5 (or any other threshold) then call the update endpoint

This will fix this by forcing the google devices to never exceed this threshold thus the user will not be billed for them

The downside of this is that if a device does not send at least 5 requests then it cannot be overwritten in the dashboard. Also if app calls setChannel then we need to ensure that it will work even if there is not such device in the supabase db.
setChannel is a gigantic pain when caching

rishi-raj-jain · 2023-09-02T18:43:27Z

@WcaleNieWolny isn't that the same as I've been suggesting in #267 to @riderx?

WcaleNieWolny · 2023-09-02T18:47:12Z

Perhaps, I was not sure what "forever blocked" ment. I proposed this as I submitted a PR with redis and it was just a thought

rishi-raj-jain · 2023-09-02T18:59:16Z

@riderx mentions in #267 (comment) that redis isn't an option, but anyways he refers to storing lot of IPs as an expensive operation. In any case, of rate limiting by 5 attempts or rate limiting by 1 attempt (that is within the last time vs new time wherever the user should be able to update), we're storing IPs with unique param (say app_id) of each update request that comes. But if Redis becomes part of the stack, the cost of Upstash is like 0.2$ per 100K which I think would be fine for @riderx?

rishi-raj-jain · 2023-09-02T18:59:59Z

Perhaps, I was not sure what "forever blocked" ment. I proposed this as I submitted a PR with redis and it was just a thought

Hope you didn't take me the wrong way, I was just confirming

WcaleNieWolny · 2023-09-02T19:09:31Z

that redis isn't an option

A lot of things happened after that comment. Capgo got a gigantic client that caused a lot of downtime as capgo was not able to scale. Since then adding redis became an option.

I already implemented caching for /updates and I wanted to build upon that. This implementation works with device id and NOT the IP. I believe using this device_id is enough. Why would we store the ip if we can just respond with the cached response without ever calling the updates endpoint for new devices that have not yer made at least 5 requests? This would essentially create zero records in supabase and remove the unnecessary cost from the users bill.

Regardless, I am sorry if I appeared as if I was trying to steal your idea

rishi-raj-jain · 2023-09-02T19:16:31Z

Regardless, I am sorry if I appeared as if I was trying to steal your idea

Nah, that's why I clarified here at #229 (comment)

rishi-raj-jain · 2023-09-02T19:16:42Z

Go ahead, rock it!

riderx · 2023-09-03T00:11:01Z

Thanks guys for putting your 2 brain on it !
I value your opinion but I wonder if such practices ( the 5 first request idea )
Will not reduce the accuracy of stats about usage, and also bring weird behavior for set channel like happening too late when user expect instant update.
For me the best idea for Google issue is more finding a way to identify and ban the right IP

rishi-raj-jain · 2023-09-04T08:05:27Z

Yeah, for that, @riderx we can tweak my solution of storing into redis as 1 or forever flag to storing the "hot" IPs by using redis keys with expiry. Shall I elaborate more?

rishi-raj-jain · 2023-09-04T08:08:26Z

So imagine only 10 hot IPs and 1M not so hot IPs,

We store all of them in the redis with set to expiry say X (an inbuilt functionality of redis). If the time crosses X and we see no re-occurence the key is dropped automatically from your database. If we do receive, it's kept in your database for an increased number of the count.

This takes care of the database maintaining itself.

Also, note, we have to decide what is considered as a hot IP. Let's say if any IP gets above Y updates refs in the period X, we'll blacklist it.

rishi-raj-jain · 2023-09-04T08:09:26Z

That's what I meant, let me know if this makes sense or needs a POC, I'm good at those :D

WcaleNieWolny · 2023-09-04T08:11:00Z

There is no way to do a PoC right now, redis is still not merged

WcaleNieWolny · 2023-09-04T08:11:33Z

You could expand on my redis branch but right now I am not sure when will it get merged

rishi-raj-jain · 2023-09-04T08:17:44Z

I mean we don't need the whole redis PR to be merged for this, this'll be super quick with just using the upstash package in the update endpoint.

WcaleNieWolny · 2023-09-04T08:21:29Z

Okay, go for the PoC 👍

rishi-raj-jain · 2023-09-04T22:32:14Z

Awesome! I'll be on it in a day.

ologbonowiwi · 2023-10-13T19:19:07Z

What still needs to be done on this one?

I'm up to it @WcaleNieWolny @riderx.

/attempt #229

Options

Cancel my attempt

WcaleNieWolny · 2023-10-13T19:24:22Z

Mainly a actual test run with a published play store app if I understand correctly

ologbonowiwi · 2023-10-13T19:25:24Z

So we're already blocking it, we only need to check if the block is working?

rishi-raj-jain · 2023-10-13T19:37:38Z

I'm on it! Had a discussion with @riderx where I need to add an endpoint for this

rishi-raj-jain · 2023-10-13T19:37:47Z

On it after the CF thing we're doing today

ologbonowiwi · 2023-10-13T19:42:17Z

Oh sorry for it, your pr was closed I thought you canceled your attempt. Go for it mate!

rishi-raj-jain · 2023-10-13T19:43:41Z

wait what

rishi-raj-jain · 2023-10-13T19:43:53Z

omg I deleted the cap-go/capgo for a fork shit

rishi-raj-jain · 2023-10-13T19:44:06Z

that caused it, will re-open cc @riderx

rishi-raj-jain · 2023-10-13T20:18:10Z

appreciate your understanding @ologbonowiwi!

algora-pbc · 2023-10-18T19:19:22Z

@ologbonowiwi: Reminder that in 5 days the bounty will become up for grabs, so please submit a pull request before then 🙏

algora-pbc · 2023-10-18T19:24:38Z

@WcaleNieWolny: Reminder that in 5 days the bounty will become up for grabs, so please submit a pull request before then 🙏

rishi-raj-jain · 2023-11-16T21:13:45Z

@ologbonowiwi @WcaleNieWolny please go ahead and crush this, I'm running out of bandwidth right now.

ologbonowiwi · 2023-11-16T21:20:36Z

@WcaleNieWolny @riderx folks, do you have bandwidth to review/merge if I open a PR to it?

If so, I'd be happy to tackle this

WcaleNieWolny · 2023-11-18T05:01:26Z

I do have the bandwith, but I do not own a google play app so I would be unable to test this. If you can present something sensible then yeah, this could work but be aware that testing if your solution works might be difficult / require martin's help

riderx · 2023-11-18T14:36:41Z

I have set capgo app to use the debug endpoint and I have the headers the Google play phone sent, outside of IP block I have no better idea, I can share them if needed

ologbonowiwi · 2023-11-18T16:02:56Z

I'm thinking about pulling the changes from #267 and handling the comments made, and then you folks can review it again.

Sounds good @WcaleNieWolny @riderx?

WcaleNieWolny · 2023-11-18T19:09:13Z

Nah, I do not think I will work

WcaleNieWolny · 2023-11-18T19:14:23Z

The real solution could be something like:

Add a new block flag in capgo that blocks the update endpoint for a device with this version. (if a device has the build in version equal to the blocked version just never update)
Figure out what happens when user upload app to google store (I do not own a google store app and neither do you so this MIGHT be expensive (25 USD if i understand correctly)
Listen to an event when the google devices are finished testing (if such event exists, that I do not know) OR
Create a cron job that will undo this block flag after an hour (So after an hour this flag should be disabled, then you can let the app owner know it is ready for deployment or sth)

Tho you might want to ask martin if he likes this idea

rishi-raj-jain · 2023-11-18T19:17:09Z

@WcaleNieWolny

The super easy way to approach this is to have an app such as capgo's own app to make POST requests to a specific endpoint and capture all the data from the request, namely IP and headers. Can add additional data to the request based on whatever Martin's ready to send with the request from the app.

Observe for 3 days after releasing multiple versions.

All this with only the app, we've access to! Not for all the customers or our actual runtime. Just to see where the update requests come from with all the data we need to safely distinguish.

Thank me with a tip later.

riderx · 2023-11-20T19:14:12Z

The Capgo app is already available in Google Play Store and I already receive many IPS of Google test devices who send to test endpoint updates_debug.
https://play.google.com/store/apps/details?id=ee.forgr.capacitor_go

That what i tried to share on my last message.
here is some data:

{
  "accept-encoding": "gzip",
  "cdn-loop": "cloudflare; subreqs=1",
  "cf-connecting-ip": "70.32.128.247",
  "cf-ew-via": "15",
  "cf-ray": "82927c4a50132bf4-ORD",
  "cf-visitor": '{"scheme":"https"}',
  "cf-worker": "supabase.co",
  "content-length": "280",
  "content-type": "application/json; charset=utf-8",
  host: "edge-runtime.supabase.com",
  "user-agent": "Dalvik/2.1.0 (Linux; U; Android UpsideDownCake Build/UPB2.230407.014)",
  "x-amzn-trace-id": "Root=1-655b9a58-7682c01f38684d1a5461e84a",
  "x-deno-subhost": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImtpZCI6InN1cGFiYXNlIn0.eyJkZXBsb3ltZW50X2lkIjoieHZ3enBvYXpteGtx"... 224 more characters,
  "x-forwarded-for": "70.32.128.247,70.32.128.247, 3.2.50.150",
  "x-forwarded-host": "edge-runtime.supabase.com",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
},
{
  "accept-encoding": "gzip",
  "cdn-loop": "cloudflare; subreqs=1",
  "cf-connecting-ip": "74.125.210.133",
  "cf-ew-via": "15",
  "cf-ray": "82927c4a17db07ee-ATL",
  "cf-visitor": '{"scheme":"https"}',
  "cf-worker": "supabase.co",
  "content-length": "283",
  "content-type": "application/json; charset=utf-8",
  host: "edge-runtime.supabase.com",
  "user-agent": "Dalvik/2.1.0 (Linux; U; Android 6.0.1; Nexus 5X Build/MTC20F)",
  "x-amzn-trace-id": "Root=1-655b9a58-023df6c15a1182f239e329c0",
  "x-deno-subhost": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImtpZCI6InN1cGFiYXNlIn0.eyJkZXBsb3ltZW50X2lkIjoieHZ3enBvYXpteGtx"... 224 more characters,
  "x-forwarded-for": "74.125.210.133,74.125.210.133, 99.83.101.75",
  "x-forwarded-host": "edge-runtime.supabase.com",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
},
{
  "accept-encoding": "gzip",
  "cdn-loop": "cloudflare; subreqs=1",
  "cf-connecting-ip": "74.125.210.134",
  "cf-ew-via": "15",
  "cf-ray": "82927c4147a74523-ATL",
  "cf-visitor": '{"scheme":"https"}',
  "cf-worker": "supabase.co",
  "content-length": "283",
  "content-type": "application/json; charset=utf-8",
  host: "edge-runtime.supabase.com",
  "user-agent": "Dalvik/2.1.0 (Linux; U; Android 6.0.1; Nexus 5X Build/MTC20F)",
  "x-amzn-trace-id": "Root=1-655b9a57-6fac478c3a0614f548713a0b",
  "x-deno-subhost": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImtpZCI6InN1cGFiYXNlIn0.eyJkZXBsb3ltZW50X2lkIjoieHZ3enBvYXpteGtx"... 224 more characters,
  "x-forwarded-for": "74.125.210.134,74.125.210.134, 99.83.101.73",
  "x-forwarded-host": "edge-runtime.supabase.com",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
},
{
  "accept-encoding": "gzip",
  "cdn-loop": "cloudflare; subreqs=1",
  "cf-connecting-ip": "70.32.128.247",
  "cf-ew-via": "15",
  "cf-ray": "82927c1187d02bf4-ORD",
  "cf-visitor": '{"scheme":"https"}',
  "cf-worker": "supabase.co",
  "content-length": "280",
  "content-type": "application/json; charset=utf-8",
  host: "edge-runtime.supabase.com",
  "user-agent": "Dalvik/2.1.0 (Linux; U; Android UpsideDownCake Build/UPB2.230407.014)",
  "x-amzn-trace-id": "Root=1-655b9a4f-15e9292359ef0acf0f522158",
  "x-deno-subhost": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImtpZCI6InN1cGFiYXNlIn0.eyJkZXBsb3ltZW50X2lkIjoieHZ3enBvYXpteGtx"... 224 more characters,
  "x-forwarded-for": "70.32.128.247,70.32.128.247, 3.2.50.150",
  "x-forwarded-host": "edge-runtime.supabase.com",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
},
{
  "accept-encoding": "gzip",
  "cdn-loop": "cloudflare; subreqs=1",
  "cf-connecting-ip": "70.32.128.247",
  "cf-ew-via": "15",
  "cf-ray": "82927bebd6432bf4-ORD",
  "cf-visitor": '{"scheme":"https"}',
  "cf-worker": "supabase.co",
  "content-length": "280",
  "content-type": "application/json; charset=utf-8",
  host: "edge-runtime.supabase.com",
  "user-agent": "Dalvik/2.1.0 (Linux; U; Android UpsideDownCake Build/UPB2.230407.014)",
  "x-amzn-trace-id": "Root=1-655b9a49-5beb99144ec3e9bd1960a6a6",
  "x-deno-subhost": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImtpZCI6InN1cGFiYXNlIn0.eyJkZXBsb3ltZW50X2lkIjoieHZ3enBvYXpteGtx"... 224 more characters,
  "x-forwarded-for": "70.32.128.247,70.32.128.247, 3.2.50.146",
  "x-forwarded-host": "edge-runtime.supabase.com",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
},
{
  "accept-encoding": "gzip",
  "cdn-loop": "cloudflare; subreqs=1",
  "cf-connecting-ip": "70.32.128.247",
  "cf-ew-via": "15",
  "cf-ray": "82927ba503d52bf4-ORD",
  "cf-visitor": '{"scheme":"https"}',
  "cf-worker": "supabase.co",
  "content-length": "280",
  "content-type": "application/json; charset=utf-8",
  host: "edge-runtime.supabase.com",
  "user-agent": "Dalvik/2.1.0 (Linux; U; Android UpsideDownCake Build/UPB2.230407.014)",
  "x-amzn-trace-id": "Root=1-655b9a3e-0666a1fa4cba406d3f65de55",
  "x-deno-subhost": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImtpZCI6InN1cGFiYXNlIn0.eyJkZXBsb3ltZW50X2lkIjoieHZ3enBvYXpteGtx"... 224 more characters,
  "x-forwarded-for": "70.32.128.247,70.32.128.247, 3.2.50.105",
  "x-forwarded-host": "edge-runtime.supabase.com",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https"
}

rishi-raj-jain · 2023-11-20T19:17:40Z

@riderx these were inbound to your infra from your app?

rishi-raj-jain · 2023-11-20T19:21:17Z

@riderx were these received only when the app is being updated by Google?

riderx · 2023-11-20T20:58:48Z

yes from google cloud device tester to capgo backend only

algora-pbc bot added the 💎 Bounty label Aug 9, 2023

rishi-raj-jain mentioned this issue Aug 14, 2023

fix: Find a better way to block google play test device #267

Closed

riderx mentioned this issue Aug 23, 2023

[Snyk] Upgrade vue-demi from 0.14.0 to 0.14.5 #301

Closed

Find a better way to block google play test device #229

Find a better way to block google play test device #229

Comments

riderx commented May 11, 2023

riderx commented Aug 9, 2023

algora-pbc bot commented Aug 9, 2023 • edited Loading

💎 $100 bounty • Capgo

Steps to solve:

Additional notes:

rishi-raj-jain commented Aug 13, 2023 • edited by algora-pbc bot Loading

riderx commented Aug 14, 2023

rishi-raj-jain commented Aug 14, 2023 • edited Loading

WcaleNieWolny commented Sep 2, 2023 • edited Loading

rishi-raj-jain commented Sep 2, 2023

WcaleNieWolny commented Sep 2, 2023

rishi-raj-jain commented Sep 2, 2023

rishi-raj-jain commented Sep 2, 2023 • edited Loading

WcaleNieWolny commented Sep 2, 2023 • edited Loading

rishi-raj-jain commented Sep 2, 2023

rishi-raj-jain commented Sep 2, 2023 • edited Loading

riderx commented Sep 3, 2023

rishi-raj-jain commented Sep 4, 2023

rishi-raj-jain commented Sep 4, 2023 • edited Loading

rishi-raj-jain commented Sep 4, 2023

WcaleNieWolny commented Sep 4, 2023

WcaleNieWolny commented Sep 4, 2023

rishi-raj-jain commented Sep 4, 2023 • edited Loading

WcaleNieWolny commented Sep 4, 2023

rishi-raj-jain commented Sep 4, 2023

ologbonowiwi commented Oct 13, 2023 • edited by algora-pbc bot Loading

WcaleNieWolny commented Oct 13, 2023 via email • edited Loading

ologbonowiwi commented Oct 13, 2023

rishi-raj-jain commented Oct 13, 2023

rishi-raj-jain commented Oct 13, 2023

ologbonowiwi commented Oct 13, 2023 via email

rishi-raj-jain commented Oct 13, 2023

rishi-raj-jain commented Oct 13, 2023

rishi-raj-jain commented Oct 13, 2023

rishi-raj-jain commented Oct 13, 2023

algora-pbc bot commented Oct 18, 2023

algora-pbc bot commented Oct 18, 2023

rishi-raj-jain commented Nov 16, 2023

ologbonowiwi commented Nov 16, 2023

WcaleNieWolny commented Nov 18, 2023

riderx commented Nov 18, 2023

ologbonowiwi commented Nov 18, 2023

WcaleNieWolny commented Nov 18, 2023

WcaleNieWolny commented Nov 18, 2023

rishi-raj-jain commented Nov 18, 2023 • edited Loading

riderx commented Nov 20, 2023

rishi-raj-jain commented Nov 20, 2023

rishi-raj-jain commented Nov 20, 2023

riderx commented Nov 20, 2023

algora-pbc bot commented Aug 9, 2023 •

edited

Loading

rishi-raj-jain commented Aug 13, 2023 •

edited by algora-pbc bot

Loading

rishi-raj-jain commented Aug 14, 2023 •

edited

Loading

WcaleNieWolny commented Sep 2, 2023 •

edited

Loading

rishi-raj-jain commented Sep 2, 2023 •

edited

Loading

WcaleNieWolny commented Sep 2, 2023 •

edited

Loading

rishi-raj-jain commented Sep 2, 2023 •

edited

Loading

rishi-raj-jain commented Sep 4, 2023 •

edited

Loading

rishi-raj-jain commented Sep 4, 2023 •

edited

Loading

ologbonowiwi commented Oct 13, 2023 •

edited by algora-pbc bot

Loading

WcaleNieWolny commented Oct 13, 2023 via email •

edited

Loading

rishi-raj-jain commented Nov 18, 2023 •

edited

Loading