GHES instance with internal certificate throwing "unknown authority" error

[Hdom]

Hello, we are trying to set up a garm instance connected to an internal GitHub Enterprise Server instance and garm is unable to connect to the github instance due to an unknown authority tls error.

error:

time=2025-05-02T21:35:06.844Z level=INFO source=/workspace/runner/pool/pool.go:1599 msg="running initial tool update"
time=2025-05-02T21:35:06.894Z level=ERROR source=/workspace/runner/pool/pool.go:404 msg="failed to update tools for entity" error="Get \"https://github.internal/api/v3/repos/ORG/repo/actions/runners/downloads\": could not refresh installation id 2's token: could not get access_tokens from GitHub API for installation ID 2: tls: failed to verify certificate: x509: certificate signed by unknown authority\nfetching runner tools\ngithub.com/cloudbase/garm/runner/pool.(*basePoolManager).FetchTools\n\t/workspace/runner/pool/pool.go:1999\ngithub.com/cloudbase/garm/runner/pool.(*basePoolManager).updateTools\n\t/workspace/runner/pool/pool.go:402\ngithub.com/cloudbase/garm/runner/pool.(*basePoolManager).Start.func1\n\t/workspace/runner/pool/pool.go:1600\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700" entity=UKG-FedRAMP/bst-garm-config
time=2025-05-02T21:35:06.894Z level=ERROR source=/workspace/runner/pool/pool.go:1601 msg="failed to update tools" error="failed to update tools for entity ORG/repo: fetching runner tools: Get \"https://github.internal/api/v3/repos/ORG/repo/actions/runners/downloads\": could not refresh installation id 2's token: could not get access_tokens from GitHub API for installation ID 2: tls: failed to verify certificate: x509: certificate signed by unknown authority"

I have tried everything I could think of:

• Added Root CA cert, Intermediate CA cert, and host cert to /usr/local/share/ca-certificates and ran update-ca-certificates.
• Verified that the certificates are present in /etc/ssl/certs once update-ca-certificates is ran.
• Verified I can curl the instance from within the machine running garm without any issues, certificate is trusted.
• Tried adding the certs to /usr/share/ca-certificates and updating /etc/ca-certificates.conf as advised by this https://stackoverflow.com/a/74575551.
• Tried setting ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt and ENV SSL_CERT_DIR=/etc/ssl/certs together and separately. I also tried setting the environment variables directly in the machine (export ...) and directly before executing the garm service (SSL_CERT_DIR=/etc/ssl/certs ./garm -config config.toml).
• Tried running with and without root permissions.

Runtime details:
Runtime Image: library/ubuntu:22.04
Additional Packages: gettext-base, wget, apt-transport-https, ca-certificates, gnupg, curl, jq, google-cloud-cli, kubectl, google-cloud-sdk-gke-gcloud-auth-plugin, yq

GARM Binary:
Custom garm binary based off v0.1.5.
Builder: Google Cloud Build.
Build Image: golang:1.24-bookworm
Build command:

go build -o $_FILE_NAME \
      -tags osusergo,netgo,sqlite_omit_load_extension \
      -ldflags "-linkmode external -extldflags '-static' -s -w -X main.Version=$(git describe --tags --match='v[0-9]*' --dirty --always)" \
      ./cmd/garm

Any ideas as to what might be causing this?
Could it be an incompatibility between the binary build in debian and running in ubuntu?

[gabriel-samfira]

Hi @Hdom !

Try:

garm-cli github endpoint update --ca-cert-path /your/ghes/ca.pem ghes.example.com

See:

https://github.com/cloudbase/garm/blob/main/doc/using_garm.md#creating-a-github-endpoint

Edit:

The ca you specify here will be used to create the github client and will be passed along to the runners as an additional CA, so that your runners can also communicate with ghes.

[Hdom]

Hello @gabriel-samfira, I had made sure to add the bundle before, still getting same error, here are the details:

• Downloaded cert chain using </dev/null openssl s_client -connect github.internal:443 -showcerts | sed -n '/-----BEGIN/,/-----END/p' > bundle.pem
• Updated endpoint garm-cli github endpoint update github.internal --ca-cert-path bundle.pem
• There are 3 certs in the chain so I tried each individual one separately too.

[gabriel-samfira]

Out of curiosity, could you try using a PAT instead of an app just to see if the error goes away?

This is where we compose the transport for the github client:

garm/params/params.go

Lines 635 to 687 in 0d53dce

	func (g GithubCredentials) GetHTTPClient(ctx context.Context) (*http.Client, error) {
	var roots *x509.CertPool
	if g.CABundle != nil {
	roots = x509.NewCertPool()
	ok := roots.AppendCertsFromPEM(g.CABundle)
	if !ok {
	return nil, fmt.Errorf("failed to parse CA cert")
	}
	}
	httpTransport := &http.Transport{
	TLSClientConfig: &tls.Config{
	RootCAs: roots,
	MinVersion: tls.VersionTLS12,
	},
	}
	var tc *http.Client
	switch g.AuthType {
	case GithubAuthTypeApp:
	var app GithubApp
	if err := json.Unmarshal(g.CredentialsPayload, &app); err != nil {
	return nil, fmt.Errorf("failed to unmarshal github app credentials: %w", err)
	}
	if app.AppID == 0 || app.InstallationID == 0 || len(app.PrivateKeyBytes) == 0 {
	return nil, fmt.Errorf("github app credentials are missing required fields")
	}
	itr, err := ghinstallation.New(httpTransport, app.AppID, app.InstallationID, app.PrivateKeyBytes)
	if err != nil {
	return nil, fmt.Errorf("failed to create github app installation transport: %w", err)
	}
	tc = &http.Client{Transport: itr}
	default:
	var pat GithubPAT
	if err := json.Unmarshal(g.CredentialsPayload, &pat); err != nil {
	return nil, fmt.Errorf("failed to unmarshal github app credentials: %w", err)
	}
	httpClient := &http.Client{Transport: httpTransport}
	ctx = context.WithValue(ctx, oauth2.HTTPClient, httpClient)
	if pat.OAuth2Token == "" {
	return nil, fmt.Errorf("github credentials are missing the OAuth2 token")
	}
	ts := oauth2.StaticTokenSource(
	&oauth2.Token{AccessToken: pat.OAuth2Token},
	)
	tc = oauth2.NewClient(ctx, ts)
	}
	return tc, nil
	}

Apps and PATs have different code paths. For apps we leverage another package and I want to verify if it's because I don't properly pass in the CA certs to that package or if the issue is in the way we create the transport and pass it along to the github client.

Sadly, I do not have a GHES to test against, so I may need to bug you to verify potential fixes.

[gabriel-samfira]

I think I may know where the issue is. Will propose a PR soon that you can test out.

[gabriel-samfira]

But do try out PATs in the meantime.

[Hdom]

@gabriel-samfira I apologize for the delay, I switched to a pat and it was able to update tools

time=2025-05-03T21:29:56.760Z level=INFO source=garm/runner/pool/pool.go:1599 msg="running initial tool update"
time=2025-05-03T21:29:57.018Z level=DEBUG source=garm/runner/pool/pool.go:413 msg="successfully updated tools" pool_mgr=ORG/repo pool_type=repository

[gabriel-samfira]

okay. Then I need to focus on the ghinstallation bit and determine how to properly pass in the transport which includes the root ca pool.

[Hdom]

I am available to verify potential fixes, just let me know

[gabriel-samfira]

Count you try out:

• set base URL in ghinstallation transport #382

[Hdom]

It stopped throwing the unknown authority error which I see as progress but getting new error which I think its related to redirection

time=2025-05-03T22:04:16.048Z level=ERROR source=garm/runner/pool/pool.go:404 msg="failed to update tools for entity" error="Get \"https://github.internal/api/v3/repos/ORG/repo/actions/runners/downloads\": could not refresh installation id 2's token: received non 2xx response status \"302 Found\" when fetching https://github.internal/app/installations/2/access_tokens\nfetching runner tools\ngithub.com/cloudbase/garm/runner/pool.(*basePoolManager).FetchTools\n\tgarm/runner/pool/pool.go:1999\ngithub.com/cloudbase/garm/runner/pool.(*basePoolManager).updateTools\n\tgarm/runner/pool/pool.go:402\ngithub.com/cloudbase/garm/runner/pool.(*basePoolManager).Start.func1\n\tgarm/runner/pool/pool.go:1600\nruntime.goexit\n\t/home/dennyso/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.3.linux-amd64/src/runtime/asm_amd64.s:1695" entity=ORG/repo
time=2025-05-03T22:04:16.048Z level=ERROR source=garm/runner/pool/pool.go:1601 msg="failed to update tools" error="failed to update tools for entity ORG/repo: fetching runner tools: Get \"https://github.internal/api/v3/repos/ORG/repo/actions/runners/downloads\": could not refresh installation id 2's token: received non 2xx response status \"302 Found\" when fetching https://github.internal/app/installations/2/access_tokens"

[Hdom]

The access_tokens url seems to be missing the api part:
https://github.internal/app/installations/2/access_tokens
shoudl be
https://github.internal/api/v3/app/installations/2/access_tokens
https://docs.github.com/en/enterprise-server@3.16/rest/apps/apps?apiVersion=2022-11-28#create-an-installation-access-token-for-an-app

[Hdom]

I believe APIBaseURL are values configured in the github endpoint, mine currently look like this:

+----------------+------------------------------------------------------------------+
| FIELD          | VALUE                                                            |
+----------------+------------------------------------------------------------------+
| Name           | github.internal                                                  |
| Base URL       | https://github.internal                                          |
| Upload URL     | https://github.internal                                          |
| API Base URL   | https://github.internal                                          |

Im going to try updating that value to include the api/v3

[gabriel-samfira]

the the go-github client should do that automatically.

[Hdom]

the the go-github client should do that automatically.

That may be so but it did fix the issue, tools are updating successfully now, not sure what other unintended consequences it might bring

[gabriel-samfira]

there shouldn't be any. the URLs are only used in the github client. The BaseURL should be left as https://github.internal if that is the URL used to access the UI. We use that to determine the origin endpoint of the jobs. The API Base URL and Upload URL are used in the client and aparently should have the /api/v3 URI. Could you also please test that PATs still work after the change?