Repository navigation
Add support for large packages in go_repos - #320
Conversation
| complete_flag = '-complete ' if complete else '' | ||
| embed_flag = ' -embedcfg $SRCS_EMBED' if embedcfg else '' | ||
| out_cmd = ' -o "$OUTS_O" -symabis $SRCS_ABI -asmhdr "$OUTS_H"' if abi else ' -o "$OUT"' | ||
| srcs_var = '$(cat _plz/named_srcs/go)' if large_package else '$SRCS_GO' |
There was a problem hiding this comment.
This method solves the immediate problem (i.e. making github.com/elastic/go-elasticsearch/v9 compilable), but it will still cause the shell to feed the contents of _plz/named_srcs/go into the arguments list for please_go filter, which risks exceeding ARG_MAX if the number of files/length of the file names is sufficiently large (something that's more likely given that this feature will inherently be used for packages containing large numbers of files). I think it'd be better for please_go filter to consume the list of srcs from a file or a file descriptor, which would allow for more efficient argument-passing:
# Regular package:
echo "$SRCS_GO" | $TOOLS_PLEASE_GO filter {build_tags} -
# Large package:
$TOOLS_PLEASE_GO filter {build_tags} _plz/named_srcs/go
There was a problem hiding this comment.
The solution you propose doesn't help much, because we're still capturing the output of the filter step and passing it in to the compilation command. At least we're nolonger doing so twice (once as an env var and once in argv).
I'll merge this as is, and file a ticket to follow up.
There was a problem hiding this comment.
Actually, as far as I can see https://pkg.go.dev/cmd/compile only accepts a list of input files as argv, so I don't think we can do anything better here
Co-authored-by: Chris Novakovic <chris@chrisn.me.uk>
2 participants
This allows using packages like
github.com/elastic/go-elasticsearch/v9/typedapi/typeswhich contains a truly impressive 1800+ files