Deprecation of GitHub teams API URL format that we use

[jtgeibel]

While investigating #2617, I noticed a response header linking to this blog post about deprecation of the API URL we are using. It looks like we have until February 1st, 2021 to migrate to the new API path.

• Legacy URL - here is our existing usage of that API.
• New API, scoped by org

There are 2 forms of the new URL. The slug version GET /orgs/:org/teams/:team_slug/memberships/:username uses the org and team slugs, which we currently store in the database as teams.login in the form github:org:team. However my understanding is that with this format we'll probably also want to issue a separate API call to ensure teams.github_id matches the current team.

For the id based format, GET /organizations/:org_id/team/:team_id/memberships/:username, I don't believe that we currently track the org_id anywhere. Perhaps we could start tracking this and use the existing API (before it goes away) to backfill all the org_ids in the database.

[StylusEater]

• start tracking the org for each team (approximately 259 teams at the moment)
• add a bin we can run on Heroku to backfill the data while respecting rate limits in github
• 2nd PR to start using the new API once we have the org data
• rely on the ID version, let's not use the plaintext names

[thiagoarrais]

I'd like to tackle this in case anyone else isn't trying already. I can dedicate some time on an off over the coming weeks. I'm sure it's doable before the deprecation date.

I think I'll start by writing a PR for collecting the org_id when the teams are inserted into the database. Then we can work on backfilling the existing teams. Is that ok?

[StylusEater]

@thiagoarrais I'm happy to work on this with you. I too planned to start work on it. Let me know your thoughts. I'm located in US/Eastern and could work with you over Discord this weekend.

[thiagoarrais]

Nice, StylusEater. I can't work on the weekends, though. Maybe we can work together but asynchronously?

Anyway, I've opened a (draft) PR for the initial change. I'd really love to get some feedback on it. Tests are failing locally, but I'm not sure if this isn't a configuration issue.

Another possible course of action you can take is writing the backfill procedure based on this PR.

[thiagoarrais]

Quick update:

1. Removed the draft marker from the first PR
2. Started a new branch for the final code. Will turn it into a PR after the org ids are populated in the db.

[thiagoarrais]

Question: regarding the db backfill, as I understand we need a user token in order to query the GitHub API (/teams/{team_id}). May there be teams for which there are no valid tokens?

Edit: there is no need for a user token for the /teams/{team_id} endpoint. GitHub exposes the organization ID as part of the deprecation notice when this legacy endpoint is hit. It comes out as part of the link response header. I guess we can parse it out. Here is an example for the github:rust-lang-nursery:chalk team:

GET https://api.github.com/teams/2874034

HTTP/1.1 404 Not Found
link: <https://docs.github.com/changes/2020-01-21-moving-the-team-api-endpoints/>; rel="deprecation"; type="text/html", <https://api.github.com/organizations/14631425/team/2874034>; rel="alternate"

But some teams don't turn up valid organization ids, like github:rust-lang:log, for instance:

GET https://api.github.com/teams/1388257

HTTP/1.1 404 Not Found
link: <https://docs.github.com/changes/2020-01-21-moving-the-team-api-endpoints/>; rel="deprecation"; type="text/html", <https://api.github.com/organizations/0/team/0>; rel="alternate"

I guess those are already removed from GitHub? In that case the teams table will not be able to have an ord_id for every record. Is that a problem?

[thiagoarrais]

Quick update no. 2: wrote a backfilling procedure.