proxy.golang.org: Unusual traffic to git hosting service from Go

[ddevault]

Following up from #44468: I run a git hosting service which has, in the past few weeks, received elevated levels of traffic from Google-owned IP addresses performing git clones. The shape of the requests is something like this:

74.125.182.164 - - [23/Feb/2021:00:41:34 +0000] "GET /~yoink00/zaplog/info/refs?service=git-upload-pack HTTP/2.0" 200 553 "-" "git/2.30.0" "-"
74.125.182.161 - - [23/Feb/2021:00:41:34 +0000] "GET /~yoink00/zaplog/info/refs?service=git-upload-pack HTTP/2.0" 200 553 "-" "git/2.30.0" "-"
74.125.182.161 - - [23/Feb/2021:00:41:34 +0000] "POST /~yoink00/zaplog/git-upload-pack HTTP/2.0" 200 56 "-" "git/2.30.0" "-"
74.125.182.161 - - [23/Feb/2021:00:41:34 +0000] "POST /~yoink00/zaplog/git-upload-pack HTTP/2.0" 200 9434 "-" "git/2.30.0" "-"

What is the purpose of this traffic?

If it's crawling, it should set an appropriate user-agent and respect robots.txt. The traffic is coming in at a rate which I would not consider reasonable for a crawler, up to several times per second - and git clones are more expensive than other HTTP requests.

[ianlancetaylor]

CC @hyangah @heschik

[katiehockman]

Hi @ddevault

Thanks for filing the issue.

What is the purpose of this traffic?

There could be several sources of this traffic, or a combination of sources, so we will work to narrow this down. It sounds like this issue is focused on lowering the traffic, or at a minimum, documenting it so that the purpose of the traffic is clearer. We'll leave the User-Agent discussion in #44468. I have a few questions for you which can help us narrow down the cause.

You mentioned "in the past few weeks" that you've seen this behavior change. Was there a huge spike in traffic that started on a particular day that you know of, or was it gradual to the point where things are now? For example, was it right when Go 1.16 came out on Feb 16, or were you seeing this earlier than that? This can help us narrow down the root cause.

Are you seeing this across all of the code on git.sr.ht, or is it focused on a specific module or set of modules?

Additionally, do you have any data about the total volume of requests over a typical hour, or patterns you're seeing? You noted that the traffic you are seeing now is not reasonable. It would be helpful to hear your opinion of what an acceptable volume of requests would look like?

Some thoughts about where this traffic could be coming from:

• module resolution from the go command, which executes several fetches at once
• some new Go command behavior from 1.16
• proxy.golang.org may retry requests for transient errors
• proxy.golang.org runs regular jobs which run data refreshes of modules we already know about, e.g. what's @latest, what's the current list of versions, etc. If we do this work upfront, it makes the experience a lot faster (ie. a database lookup is a lot cheaper than a full fetch from the go command). And if we do these refreshes often enough, then this speed doesn't come at the cost of stale data. However, how and when we do these refreshes is something we may re-evaluate.

I also want to note that proxy.golang.org isn't trying to do any crawling. It fetches and pre-caches code that we can be confident that users will want, fetching and refreshing data that users have already asked for.
Even if hosting services see a large amount of traffic from proxy.golang.org, it may actually be the case that their total traffic would be higher if proxy.golang.org wasn't acting as a mirror for many users. Meaning that even though the sites see a spike in traffic from Google IPs, their total traffic volume could be less than it otherwise would be if users weren't contacting proxy.golang.org first.

[ddevault]

I'll give approximate answers, but knowing that these are the questions you want answered, I will be able to collect more specific answers when the behavior is next observable.

You mentioned "in the past few weeks" that you've seen this behavior change. Was there a huge spike in traffic that started on a particular day that you know of, or was it gradual to the point where things are now? For example, was it right when Go 1.16 came out on Feb 16, or were you seeing this earlier than that? This can help us narrow down the root cause.

This has occured on February 10th and 14th, then at least once per day from the 18th onwards. It seems likely that this co-incides with the Go 1.16 release, which notably made some major changes to how modules are used.

Are you seeing this across all of the code on git.sr.ht, or is it focused on a specific module or set of modules?

If there's any discernable pattern to the repositories affected, it's hard for me to tell. It would help if you could characterize the IP address I gave (74.125.182.164) and share a range of IPs which are also likely to be implicated in the same behavior so I can extract just the relevant part of the logs.

Additionally, do you have any data about the total volume of requests over a typical hour, or patterns you're seeing? You noted that the traffic you are seeing now is not reasonable. It would be helpful to hear your opinion of what an acceptable volume of requests would look like?

Hm, in my experience, it happens for a few tens of minutes at a rate of between 2 and 10 requests per second. Googlebot on its default settings, for example, only does one request every 5 seconds or something like that. In any case, if it's an automated process, it should be fetching robots.txt and letting me, the sysadmin of the remote host, configure its crawling parameters. The precise rate is not especially important, but I would like to tune it to a predictable value and then update our assumptions about network monitoring so we aren't getting a bunch of false alarms from Go crawling our servers.

Some thoughts about where this traffic could be coming from:

This list can be narrowed down: every request has appeared to come from a Google-owned IP address, so we can eliminate behavior from end users (and also hold Google accountable for whatever their servers are up to). I know that godoc.org has had (:frowning_face:) a crawling feature, no clue what proxy.go.org nor pkg.go.dev does.

[ddevault]

Note: it would be helpful of Go's release notes were dated.

[ALTree]

Note: it would be helpful of Go's release notes were dated.

This page: https://golang.org/doc/devel/release.html has dates for every release, if you need them.

[ddevault]

Thanks!

[katiehockman]

@ddevault Thanks for your fast response. To make it easier to figure out exactly which traffic is coming from us, we're going to go ahead and work on setting the User-Agent for proxy.golang.org requests, per #44468. Then you'll be able to more easily discern which requests are coming from us (rather than filtering by IP), and the logs will hopefully be easier to collect. It's a good thing to do either way, so we'll prioritize doing this.

I'll follow back up here once those changes are in production.

[ddevault]

Sounds good, thanks!

[ddevault]

With the new User-Agent in place, I can characterize the behavior more concretely now. Over the past hour, I've received 1,912 requests from proxy.golang.org, from IP blocks 74.125.0.0/16 and 173.194.0.0/16. The full list of requests is available here, with columns for the request IP, date, and hash of the module URL.

Redundant requests per IP address are somewhat reasonable:

https://paste.sr.ht/~sircmpwn/986d4c2e3f5909385b19adf6fa15bc789bff8708

But redundant requests across all IPs are less so:

https://paste.sr.ht/~sircmpwn/b46ad0b13e864923df80cb8e8285bf1661e6f872

There is some room for improvement here.

[ddevault]

Some specific recommendations:

• Reduce redundant requests across nodes

proxy.golang.org runs regular jobs which run data refreshes of modules we already know about, e.g. what's @latest, what's the current list of versions, etc. If we do this work upfront, it makes the experience a lot faster (ie. a database lookup is a lot cheaper than a full fetch from the go command). And if we do these refreshes often enough, then this speed doesn't come at the cost of stale data. However, how and when we do these refreshes is something we may re-evaluate.

• This process should obey robots.txt so the sysadmin can control the request rate or opt-out (for some or all packages).

I don't mind the volume if it's legitimate traffic, but an effort should be made to reduce redundant load, and to give the sysadmins some knobs to control the load.

Another question: do you make a fresh clone every time, or do you keep the repo around and fetch only the difference? You should maintain an LRU cache of clones and freshen them up on subsequent requests.

[katiehockman]

@ddevault
Thanks. We're going to take a look at reducing the load of our refresh jobs. I will respond back here with updates.

[ddevault]

Thanks. Would also appreciate answers to my more specific questions when you have the opportunity.

[katiehockman]

@ddevault
Sure, happy to answer those.

do you make a fresh clone every time, or do you keep the repo around and fetch only the difference?

The short answer is that yes we make a fresh clone every time. I want to provide more information below to clarify this:

It's not the case that a go get will always translate to a single git clone or even a single HTTP request. For example, if someone requests a new version that we don't already have, e.g. github.com/a/b/c@v0.0.1, then this may require module resolution if github.com/a/b/c is not a module. So the go command performs several requests, e.g. one for "github.com/a/b/c" then "github.com/a/b", then "github.com/a", etc.

However, we suspect that the majority of the traffic you are experiencing is not due to module resolution of new modules, but instead from our refresh jobs. These already know the resolved path, so the go command doesn't need to do this resolution. For example, go mod download module@version (which is what proxies are expected to use to fetch a module at a particular version), isn't guaranteed to perform a single HTTP request. It may do things like first request a list of all the tags of that module, then look for the go.mod file or other metadata, then do a shallow fetch to get the contents.

From proxy.golang.org's perspective, we shell out work to the go command, and it's up to the go command to decide how best to retrieve the information and pass it back to us. So the idea of keeping a cache of clones around isn't practical, nor would it help the go command.

However, something we can do right now is improve our refresh jobs to help with load, so we're going to look into that.

[ddevault]

From proxy.golang.org's perspective, we shell out work to the go command, and it's up to the go command to decide how best to retrieve the information and pass it back to us. So the idea of keeping a cache of clones around isn't practical, nor would it help the go command.

It doesn't seem reasonable to change the go command, but I would argue that proxy.golang.org is in a unique position among users of the go command, and as such it seems reasonable to suggest that it would be open to an improved implementation which better suits its unique setting. I mean, if not, Go is just shoving the complexity burden onto software hosting services, and with wasteful and redundant requests.

However, something we can do right now is improve our refresh jobs to help with load, so we're going to look into that.

Sounds like a good start.