Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "metrics" Prometheus endpoint - total requests by type and code. #417

Merged
merged 12 commits into from
May 11, 2020

Conversation

MichaelRybak
Copy link
Contributor

Add "metrics" endpoint. Provide a Prometheus counter of # of request broken down by request type and HTTP response code.

Part 1 of #357.

signer, err := signer.New(certCache, key, config.URLSet, rtvCache, certCache.IsHealthy,
overrideBaseURL, /*requireHeaders=*/!*flagDevelopment, config.ForwardedRequestHeaders)
overrideBaseURL, signerRequireHeaders, config.ForwardedRequestHeaders)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gofmt was breaking formatting here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What did gofmt break?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It breaks the inline comment (/requireHeaders=/)

Before gofmt:

	signer, err := signer.New(certCache, key, config.URLSet, rtvCache, certCache.IsHealthy,
		overrideBaseURL, /*requireHeaders=*/!*flagDevelopment, config.ForwardedRequestHeaders)

After:

	signer, err := signer.New(certCache, key, config.URLSet, rtvCache, certCache.IsHealthy,
		overrideBaseURL /*requireHeaders=*/, !*flagDevelopment, config.ForwardedRequestHeaders)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be a bug with gofmt. Although https://golang.org/doc/effective_go.html#commentary says that /* */ is mostly reserved for package comments and disabling code. I vote for making gofmt happy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SG!

errorMsg, errorCode = "500 internal error", http.StatusInternalServerError
// Validate HTTP method and params, parse params and attach them to req.
if !allowedMethods[req.Method] {
errorMsg, errorCode = "405 method not allowed", http.StatusMethodNotAllowed
} else {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this because 1) there's a reference to matchingRule below that would crash if it was nil, so this check is not sufficient to gracefully handle this hypothetical situation 2) unit tests already verify that matchingRule won't be nil 3) an assert would be nice for readability, but it's probably OK to just delete this. I'll add a comment.

MichaelRybak added a commit to MichaelRybak/amppackager that referenced this pull request May 7, 2020
MichaelRybak added a commit to MichaelRybak/amppackager that referenced this pull request May 7, 2020
Copy link
Contributor Author

@MichaelRybak MichaelRybak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated Go version in Travis.

@MichaelRybak MichaelRybak marked this pull request as ready for review May 7, 2020 19:27
@MichaelRybak MichaelRybak requested a review from twifkak May 7, 2020 19:27
Copy link
Member

@twifkak twifkak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks; all optional comments

github.com/prometheus/common v0.6.0/go.mod h1:eBmuwkDJBwy6iBfxCBob6t6dR6ENT/y+J+Zk0j9GMYc=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/procfs v0.0.3 h1:CTwfnzjQ+8dS6MhHHu4YswVAD99sL2wjPqP+VkURmKE=
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, how big is the amppkg binary after these added deps?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's 18.02M with these and 16.98M without them.

@@ -137,6 +149,8 @@ func (this *mux) ServeHTTP(resp http.ResponseWriter, req *http.Request) {
path := req.URL.EscapedPath()

// Find the first matching routing rule.
// Note that matchingRule won't be nil because the last rule in routing
// matrix has an emptry string and therefore matches any path.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth codifying this rather than needing a comment:

  1. Add a defaultHandler http.Handler field to the mux struct.
  2. Remove the "" entry from the routing matrix.
  3. Convert return404 into an http.Handler.
  4. Change the code in here. :)

Copy link
Contributor Author

@MichaelRybak MichaelRybak May 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I agree it's better to codify this. I've added a "defaultRule" field in mux to utilize the "rule" abstraction (e.g. to continue referring to "rule.prometheusLabel").

// Trigger a 404 attributed to healthz by adding an unexpected suffix to path.
pkgt.Get(t, mux, expand(`$HOST/healthzSOME_SUFFIX`))
expectedMetrics += `
total_requests_by_code_and_url{code="404",handler="healthz"} 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optionally, break each of these stanzas up into a separate test, where each one starts with a brand new mux and ends with a call to promtest.CollectAndCompare. This has a few benefits:

  1. It's easier to diagnose failures if they're more limited in scope.
  2. We won't have any false successes from some earlier action causing a later expectation to be met.

You could use suite to factor out the setup/teardown.

The benefits seem pretty minor in this case, but also the change seems pretty easy, so /shrug. Your call.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the input. I agree with the pros of singling out each scenario.

My intention was to run an "integration" test to check that when being accumulated, different metrics don't interfere with each other.

To achieve both goals I've put all the scenarios in an array and implemented running them both in isolation and altogether.

@MichaelRybak MichaelRybak merged commit 00ed240 into ampproject:master May 11, 2020
@MichaelRybak MichaelRybak deleted the metr1 branch May 15, 2020 17:55
@MichaelRybak MichaelRybak restored the metr1 branch May 15, 2020 17:56
cpapazian pushed a commit to cpapazian/amppackager that referenced this pull request May 19, 2020
…mpproject#417)

Add "metrics" endpoint. Provide a Prometheus counter of # of request broken down by request type and HTTP response code.

This partly implements request ampproject#357. More metrics to be added.
@MichaelRybak MichaelRybak deleted the metr1 branch June 1, 2020 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants