feature request: reduce dependency size

[medyagh]

this is a great lib but I was wondering if there is any room for optimization in terms of dependency size ! after adding go-getter dependency, our binary size increased by 15 mb .

related issue: kubernetes/minikube#5013

[medyagh]

I suspect you could use just go cloud or both aws and google cloud and you might not need to import both google coud go and aws sdk

currently in minikube in the top 5 dependency sizes, number 2 belongs to go-getter. https://gist.github.com/afbjorklund/635893ee8be7c6523e2c8c6f9e821572

10120	k8s.io/api/core/v1
5968	github.com/aws/aws-sdk-go/service/s3
3716	net/http
3204	runtime
3124	github.com/googleapis/gnostic/OpenAPIv2

[azr]

Hello @medyagh, weird question but the getter's detectors and getters are defaulted when not set and that is what I saw in minikube's downloader; the defaults are used.

What if you use instead:

client := &getter.Client{
	Src:      urlWithChecksum,
	Dst:      tmpDst,
	Mode:     getter.ClientModeFile,
	Options:  opts,
	Getter:   /* insert everything except aws/gcp getters */
	Detector: /* insert everything except aws/gcp detectors */
}

I have not checked but may be the go compiler will do some magic here and remove that code path and therefore remove that weight for you.

That would be nice; otherwise I think that splitting those getter/detectors into their own pkg as part of the v2 of the go-getter would be a great option.

[afbjorklund]

@azr:

I have not checked but may be the go compiler will do some magic here and remove that code path and therefore remove that weight for you.

Unfortunately the linker is not smart enough to do such dead-stripping of unused code. Adding the suggested configuration actually makes it bigger:

56088 out/minikube-linux-amd64
56096 out/minikube-linux-amd64

@@ -69,12 +69,29 @@ func (f DefaultDownloader) CacheMinikubeISOFromURL(url string) error {
 	// Predictable temp destination so that resume can function
 	tmpDst := dst + ".download"
 
+	httpGetter := &getter.HttpGetter{
+		Netrc: true,
+	}
+
 	opts := []getter.ClientOption{getter.WithProgress(DefaultProgressBar)}
 	client := &getter.Client{
 		Src:     urlWithChecksum,
 		Dst:     tmpDst,
 		Mode:    getter.ClientModeFile,
 		Options: opts,
+		Detectors: []getter.Detector{
+			new(getter.GitHubDetector),
+			new(getter.GitDetector),
+			new(getter.BitBucketDetector),
+			new(getter.FileDetector),
+		},
+		Getters: map[string]getter.Getter{
+			"file":  new(getter.FileGetter),
+			"git":   new(getter.GitGetter),
+			"hg":    new(getter.HgGetter),
+			"http":  httpGetter,
+			"https": httpGetter,
+		},
 	}
 
 	glog.Infof("full url: %s", urlWithChecksum)

That would be nice; otherwise I think that splitting those getter/detectors into their own pkg as part of the v2 of the go-getter would be a great option.

Having it in a separate package is one option, but using (optional) build tags also seem to do the trick of excluding the unused dependencies (if desired):

47124 out/minikube-linux-amd64

This saves 8M, or about 16% reduction.

[azr]

Hello there; ah too bad ! Thanks for trying this.

[medyagh]

Hi @azr I am curious, would you endorse this PR ? #207
It would add a lot of values for us, to reduce the binary size

[azr]

Hello there, I wanted to say that we have a v2 branch ( https://github.com/hashicorp/go-getter/tree/v2 ) and I think it is worth trying to split the getters and cmd into their own sub modules. I still have to learn/grasp on how everything works with modules 🙂. But I'm pinging @yujunz here that might want to do this and that opened #236 which is a duplicate.

---

Edit: Context on the v2: The Packer project which I help maintain needed some features which were breaking changes so I made them. PRs are here: #230 #231 #232

[yujunz]

I think it is worth trying to split the getters and cmd into their own sub modules

Made an attempt in #238 @azr