cmd/go: [modules + integration] use several goproxy sources simultaneously

[nim-nim]

This report is part of a series, filled at the request of @mdempsky, focused at making Go modules integrator-friendly.

Please do not close or mark it as duplicate before making sure you’ve read and understood the general context. A lot of work went into identifying problems points precisely.

Needed feature

Go needs to allow using multiple goproxy sources simultaneously (at least 2).

Constrains

• the sources may be specified in an environment variable, or a config file
• the config file is probably the best solution as URLs can get long and messy
• if using a config file, it should reside in /etc/go/something completed or masked by ~/.config/go/something on Linux systems, as per the Filesystem Hierarchy Standard and the XDG Base Directory Specification
• it is desirable for the list of goproxy sources to be hierarchical:
  • if a suitable module is found in several sources, take it from the higher-priority source
  • or even do not search for better (closer to HDR) module versions in lower priority sources, once something suitable has been found in higher-priority sources
• one source may be “Internet” (direct download)
• the “Internet” source must be optional

Motivation

In integrator workflows, a single goproxy can not be used, due to the distinction between:

• modules that passed QA in another CI/CD job and
• modules being created in this CI/CD job

For build reliability and security, the first class of modules must be deployed on a read-only goproxy. The second class however, has to be deployed on a read-write goproxy (because the aim of the CI/CD job is to create and write those modules).

[beoran]

For this use case, a singe multiplexing goproxy, that forwards to other go proxies could be implemented. Or, simply a file based proxy url in conjunction with go mod pack.

[nim-nim]

@beoran

It is intended to be used mostly with file goproxies¹ and those proxies are intended to be populated via go mod pack (#31302)

However, a single file goproxy can not be used, due to the distinction between:

• modules that passed QA in another CI/CD job and
• modules being created in this CI/CD job

For build reliability and security, the first class of modules must be deployed on a read-only goproxy. The second class however, has to be deployed on a read-write goproxy (because the aim of the CI/CD job is to create and write those modules).

Having to instantiate a go-specific proxy server in each go CI/CD job just because go tools can not read more than one directory of modules, would be complex, inefficient and fragile. As long as it's just copy files, make directory available, run directory indexing command, it's within easy capabilities of any CI/CD system. Adding a go-specific network server to the mix is not.

Remember that a lot of CI/CD systems are not go-specific. They have to since a lot of software is not written in a single language. Anything that requires go-specific processing by the CI/CD system (and is not included in CI/CD system default extension points) is causing problems.

¹ the usual case will be local file goproxy sources (because it's simpler to popular CI/CD job-specific directories than handle the security aspects of allowing access to some urls but not others). Of course, some of the file goproxies may be deployed on network filesystems (but the go command does not need to be aware or that).

[beoran]

I see, this is for use with file based goproxies. I agree that having a go mod pack command to easily populate a file based goproxy would be very useful.

I would think that the problem you are describing could be solved by having two different users, one to run the current job that creates the packed modules and that has read only permissions to the goproxy file location, and another user with write permissions to the goproxy directories who, if the build is sucessful, installs them there?

Also, go modules are in essence source only modules without binaries. I wonder, how do distribution maintainers solve this problem for a library in a different programming language, such as Ruby, which also has gem files with source only modules?

[nim-nim]

I see, this is for use with file based goproxies. I agree that having a go mod pack command to easily populate a file based goproxy would be very useful.

Thanks

I would think that the problem you are describing could be solved by having two different users, one to run the current job that creates the packed modules and that has read only permissions to the goproxy file location, and another user with write permissions to the goproxy directories who, if the build is sucessful, installs them there?

On a FHS Linux system, the read-only goproxy directory would not be just protected against writing, it would be owned by root and deployed at a fixed filesystem location. CI/CD read-write directories however can exist wherever the CI/CD job wants to create them in its own filesystem space.

So it's not just a read-only/read-write separation, you also have a strong filesystem location separation.

Also, go modules are in essence source only modules without binaries. I wonder, how do distribution maintainers solve this problem for a library in a different programming language, such as Ruby, which also has gem files with source only modules?

As far as I know the CI/CD environment works the same for other languages. It creates a contained environment, where things that are not owned by the CI/CD job are deployed in standard locations, and locked against modifications. The CI/CD job can create files and directories in its own separate read-write filesystem hierarchy.

If the CI/CD job is successful a subset of created files and directories is collected, with a mapping to canonical filesystem locations¹. Another CI/CD run can then request to use all of part of the result, that will then be exposed in the read/only filesystem space.

¹ The rpm idea on how to define the mapping is very basic, here is an empty directory, pretend it's / deploy everything you want to be reused inside this directory at the correct location relative to the fake root.

[nim-nim]

I wonder, how do distribution maintainers solve this problem for a library in a different programming language, such as Ruby, which also has gem files with source only modules?

BTW part of the motivation of the report series (especially #31300) is to help Go software benefit and catch up to the state of the art in CI/CD systems Linux side, which is evolving right now due to requests from the Rust and Golang Fedora SIGs (rpm-software-management/rpm#104 rpm-software-management/rpm#593 rpm-software-management/mock#245)

Other language SIGs like Python and Java were also involved in the design in a less direct way. A lot of our system tools use Python so anything that does not work for Python would have been DOA. Java would really need this too, but is hampered by the multiplicity of its component systems, and years of code rot (due to the "peg specific commit" / "rename and fork on change" / "never merge back" Java dev mindset). I’m pretty pessimistic of Java being able to leverage any CI/CD improvement in the short term.