-
Notifications
You must be signed in to change notification settings - Fork 17.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: Go 2: allow importing the main
function from a main
package
#65652
Comments
main
packagesmain
packages
main
packagesmain
packages
main
packagesmain
function from a main
package
I would humbly submit that this is not a Go 2 proposal, since the current standard compiler behaviour is not actually called for in the Go Programming Language Specification, and even if it was intended to be, this proposal is not a backwards compatibility breaking change to current practise. |
Can't we already effectively do that ? I mean, ok you can't literally do what you are proposing, but you could have the I imagine something like this: // cmd/go/go/go.go
package main
import gocmd "cmd/go"
func main() {
gocmd.Main()
} The examples you gave would be identical. This would require some automated rewrite step similar to what gobusybox do. |
I think it is a language change, as it defines new visibility rules for main packages (an alias Main, no other "exported" identifiers actually exported). I think this may be an unsafe thing to do in general:
While it's true that similar issues can arise with poorly written importable packages, I'd argue that the importable / unimportable separation of intent is important, and for the effort needed to fix those main packages it isn't much more work to move the code into an importable one. |
Hi @Jorropo,
I completely agree! For my own programs I also like to use the structure of a But what about third party programs, and being able to bundle them into a single monolithic binary? If they could be changed to adopt that structure, all the better. But what if that's not possible? I'd even be happy if there were some kind of compiler-specific mechanism to support this use case: go build -bundlemain cmd/go -bundlemain cmd/gofmt -bundlemain you/pkg/cmd/program ... multitool.go ... and then some kind of |
@griesemer ever wanted to support this, but @rsc denied it. I couldn't find the related issue now. |
Hi @seankhliao,
I agree that it is a language change, but I believe it could be backwards compatibility preserving change to Go 1. Maybe I am misunderstanding the meaning of "Go 2" in this context (which I thought would refer to truly breaking changes, such as if I were proposing that the I think it would be helpful to clarify the current intention of the language spec:
I completely agree! There are zero guarantees that, once your program calls into another But those issues would be the callers responsibility to sort through. It would not affect the behaviour of any existing programs. If an open source project received a bug report about "strange behaviour when importing your main" they could quite fairly say "we don't support that, won't fix". |
@pdmccormick The "Go 2" term in issue titles, as well as the v2 label, are to facilitate triage and search. They don't imply that a proposal is not backward compatible. |
Thank you for the clarification @zephyrtronium ! I appreciate it. |
Thank you all for you input. Starting from the origin point of the current
Further comments in the thread refer to solving the newly introduced restriction by reorganizing programs to split between the package Later in that thread, @ianlancetaylor commented:
CL 4126053 specifically removed a spec-level prohibition against importing a Given this, would it be worth creating a new issue proposing some kind of "busybox-style multi-program build" support in the standard compiler? Whether by permitting restricted main imports, or by some other way of invoking another programs |
As you say, this seems to be a reversal of #4210. The discussion there included the busybox case. What has changed that we should reconsider that earlier decision? |
The comment in #4210 that first mentioned Busybox-style binaries also mentions that they had the alternative remedy of splitting their But what about the case of bundling together existing & unmodified third party programs, to share the cost of common code that would otherwise be duplicated across multiple binaries? With the rise of Go-based Linux userlands such as u-root and gokrazy, a Go-based Linux distribution (or embedded system image) could pull together an entire suite of system tools and useful applications with a single I probably jumped too quickly to suggesting a specific solution (allow importing main packages again) when really the question I'm grappling with is, could the Go compiler help me in my quest to build multiple programs into a single executable, and offer a Go At the risk of jumping to solutions again, perhaps something like the combination of allowing |
It seems like this proposal hinges on a more philosophical question about the design goals of the language: Should it be possible for someone to embed someone else's standalone Go program as if it were a library in someone else's Go program without first negotiating an explicit library API with the author of the first program? Everything else in Go so far seems to have tilted toward giving authors control over their exported API: package symbols can be either exported or unexported, and the As an author of one such program that has intentionally not exported any library API, I would say that I would prefer this not to be accepted in the current form. I would not be opposed to an alternative that makes it opt-in for the author of the program being embedded, but I'm not really convinced such a thing is needed because it's already possible for a Go module to offer both a For the specific use-case of "build something like BusyBox", I would suggest that the author of such a thing ought to design an explicit API through which a Go module can expose one or more entry points with a specified signature, and then encourage authors of programs that would make sense to include in such a bundle to expose those entry points. Not all authors would want to participate in such a thing, but I expect that many would if the argument for it were compelling enough. |
Thank you @apparentlymart for the thoughtful comments. You have helped persuade me, importing In the case of the program you mentioned, the one that you did not want to expose a package-level API for: you wouldn't have any objection to it be invoked as a program though, would you? i.e. as an OS-level process, with |
Indeed, my concern is that running a program that expects to be in full control of its process as if it were a library inside another process would likely cause it to behave differently in that context, and that those different behaviors would then implicitly become part of the functionality that program is offering and create (possibly undesirable) pressure to keep it working in that context, despite the differences. If you were able to create an execution environment that is completely indistinguishable from the program being launched directly as a child process then I would not consider that to be problem, but I'm not convinced that the proposal as currently written would guarantee that outcome. From what you've described in your latest comment, I'm imagining something that is more like a replacement for the Go runtime's startup code, presumably not actually written in Go so that it can run before a Go execution environment is available, which does whatever special logic it needs to decide which of many In particular (but not limited to):
I realize that this is quite a pedantic set of requirements and that many programs would not actually be affected by differences here in practice, but nonetheless all of these things could materially affect a specific program's behavior, in ways that the author of that program may not be aware of until it's too late. |
@apparentlymart Great points, especially about the need to preserve Keeping with the idea of a program invocation style interface, what about a mechanism for restarting the current Go runtime into a different program? Similar to how an exec(3) syscall replaces the currently running OS level process with a new one. In the You raise an excellent point about multiple package versions. If a project opted-in to this kind of arrangement, and split out an importable Assuming that the bundled programs would still function correctly when their package requirements were resolved together, specifying the programs to bundle could be provided as flags to go build -bundle pkg1/cmd/foo -bundle pkg2/cmd/bar ./cmd/gopherbox/ In this program, Alternatively, at the risk of circling back to this topic, maybe a special case of main importing could be permitted, not for initializing side effects but just to declare that it should be bundled into the given build: package main
import (
"os"
"runtime"
_ "cmd/go"
)
func main() { runtime.ExecProgram("cmd/go", os.Args, os.Environ()) } This is introducing an irregularity to how imports are to be treated, and that may require a change to the Language Specification (although, as I have learned in this process, the Go language spec does not actually disallow |
Note that at least on Unix systems we do have https://pkg.go.dev/syscall#Exec . |
If I'm not mistaken, the Go runtime does not currently have a notion of multiple |
No, it does not. It always starts at |
I'm sorry for jumping around with ideas in this thread, but thank you for engaging with me on this. To summarize, I'm wondering if it would be possible to enable:
So a Go runtime level exec call, if you will, all from within the same executable and same running process. |
I think I get what you're thinking about here, and maybe the following set of analogies would both allow you to confirm whether I've understood correctly, and if so help others in this thread reach a similar point of understanding.
(I'm intending this comment only as an attempt to confirm that I'm understanding the new proposal correctly, not as any specific opinion on the new proposal.) |
In terms of a concrete example for this Busybox-style multitool executable, and possible storage savings: Two Docker plugins that are typically installed together are buildx and compose (I have no affiliation with Docker). When I build each of these, the binaries are 77 MiB and 75 MiB respectively. Next, I have locally modified each of these projects to move the contents of their primary This might be a bad idea for other reasons as discussed above, but it's basically those savings that I am chasing. I will be trying to see if the authors of the particular programs relevant to me would be willing to accept that kind of main/cmd split, but the motivation for this entire proposal was, could this be supported in the toolchain itself, to allow bundling together collections of Go programs into single executables. |
Yes, you've understood me perfectly, thank you! And you're absolutely right, the discussion has moved on quite a bit from the original proposal. I am happy to withdraw it, or have it withdrawn by someone else, if that is the appropriate course of action for this process? |
This is a great point: if any kind of runtime patching would be required to switch between programs then I'm definitely thinking that this is starting to seem like a bad idea. If the results were not the equivalent to what I did in my Docker example above, existing programs which are modified to explicitly opt-in by adding a boilerplate main/importable cmd split, and then having those program packages pulled into a new program as dependencies, if it would not be possible to get some level of compiler support for achieving that without the boilerplate, with the least possible changes to anything else in the ecosystem or runtime, then I think this idea should be abandoned. |
Importing main packages was in the distant past permitted, but the tools were changed to disallow it, and the rationale for that change applies to this proposal too. Therefore, this is a likely decline. Leaving open for four weeks for final comments. |
No further comments. |
Go Programming Experience
Experienced
Other Languages Experience
C, Python
Has this idea, or one like it, been proposed before?
No, not to my knowledge.Edit: Issue #4210, commit 679fd5b and CL 4126053 track when importing
main
became disallowed by the standard toolchain, but not explicitly prohibited in the language specification.Does this affect error handling?
No.
Is this about generics?
No.
Proposal
Currently, using the standard Go compiler, if you try to import a
main
package you will get this error:This proposal is to allow importing of
main
functions from Go programs under two conditions:main
package must be qualified with a package name, andMain
, which is an alias to themain
function inside themain
package.Motivation
The gobusybox project uses a source-to-source transformation step to bundle up many separate
package main
Go programs into a single monolithic executable. Each individual subprogram can then be separately invoked, whether by being specified as the first command-line argument, or by being invoked through a symlink (with the name of the symlink corresponding to the subprogram to invoke). This can yield significant storage savings, since all the bundled programs in one executable would share a single copy of compiled common code such as the Go runtime, standard library and third party packages. The alternative is to compile each program individually, leading to the duplication of compiled code. These savings can be very meaningful in resource constrained contexts such as embedded systems with limited storage.Further, there is now a huge ecosystem of Go programs and utilities, and many of them were not written to have any other way of importing and invoking them other than as programs via their
main
entrypoint, with theos
packageArgs
,Environ
,Stdout
,Stderr
,Stdin
being the expected process-oriented invocation interface. None the less, it would be incredibly useful to be able to import these existing programs without modification and bundle them into a single monolithic Go executable. There is no guarantee that those programs will "play nice" when invoked under different assumptions, but then there never is when using a third party package.Example
Language Spec Changes
I could not find any text in The Go Programming Language Specification (as of language version go1.22, dated Feb 6, 2024) that would suggest that
main
packages MUST NOT be importable. My first question is, is the standard compiler behaviour just an implementation choice, or does that behaviour reflect something that was actually intended to be included in the Language Specification? If the current proposal is therefore not a change to the language but rather the standard compiler, then this proposal should be recategorized.Assuming that the current behaviour was intended to be included in the Language Specification, and this proposal should be evaluated in the context as being a change to that behaviour, here are the further changes required.
In the section "Program Execution", it states:
While this statement would remain true for the actual
main
function of the program, it would not be true for amain
that was imported and invoked like a regular function. But this proposed change would not break any existing programs.In the section "Import declarations", the following addition is suggested:
In the section "Exported identifiers", adding a 3rd point to the list:
Informal Change
In Go, program execution starts by calling your
main
function inside apackage main
source file. Currently you are now allowed to import anotherpackage main
, and even if you could, you would be able to call themain
function, since by the rules of exported identifiers themain
function does not qualify because of the lowercased starting letter. But since there are so many useful Go programs out there, what if we could get access to theirmain
functions, and bundle their functionality into our own programs? Not all third party programs will work as expected this way (have you ever thought about what it would be like for someone else to call your programmain
as a function?), but compiling one big monolithic program together might be easier than having to deal with multiple executables.Is this change backward compatible?
Yes. Since importing a
main
package is currently not allowed in practise, no valid programs have been doing this. Existing programs would be unaffected.By aliasing the
main
function to an exported identifierMain
, as a special case behaviour formain
packages, the rules around exported identifiers would be unchanged.The
main
function from existing programs may not necessarily be well-behaved from the perspective of being imported and called from another. But this situation is no different than importing a third party package and finding that it, for example, callspanic
when returning an error would have been more convenient for the caller. It simply may not be convenient or practical to import another program and actually make it useful.There is no requirement on programs, new or old, to change anything about how they behave or interact with the operating system. It is entirely the callers responsibility to provide a suitable state when calling
main
(for example, by manipulatingos.Args
to still be meaningful).Orthogonality: How does this change interact or overlap with existing features?
I think it's a fairly conservative extension to current practise. Since the import must include an explicit package name, the compiler would still return an error if someone accidentally tried to import an
main
package without a package name qualifier. The user would have to be quite intentional to access this capability.Would this change make Go easier or harder to learn, and why?
While I do understand the argument against being able to import another
main
package, it always struck me as a a bit irregular that the rules around importingmain
were so different to any other package. A lot of projects put a large amount of code into theirmain
packages, because they are intending to process a program, and not a reusable/importable package. But they still intended to be invokable, as programs, which is very similar in concept to being able to call theirmain
function.With the rise of a robust module system and wider ecosystem, perhaps this restriction made more sense in an earlier era of Go's development?
But this proposal could lead to unintuitive behaviour for those who use it. Why are other seemingly legitimately exportable identifiers not in fact exported? What if a program contained both a
main
andMain
function? Only themain
function should be exported asMain
, otherwise existing programs may be unimportable (i.e. ifMain
had a different function signature, or if it was not a function at all), but this could be extremely confusing.Cost Description
I am not aware of anywhere else in Go where an identifier without an uppercase first character can be exported (i.e.
main
), or where one imported identifier is an alias for another (Main
=>main
). This would require even more special case rules for how themain
package is handled.Changes to Go ToolChain
Entire compiler, from parser to linker, plus any tool that interacts with source code (such as vet, gopls, gofmt and goimports)
Performance Costs
None (to the best of my knowledge!)
Prototype
Work in progress. I'm working this out by first relaxing this error condition and seeing what happens next.
The text was updated successfully, but these errors were encountered: