Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding gcs call log for storage layout api #2629

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

anushka567
Copy link
Member

@anushka567 anushka567 commented Oct 25, 2024

Description

As part of HNS , storage layout API is called, logs for this does not come at TRACE severity unlike other GCS calls. Hence, added the logic for the same so that when log severity is set to TRACE, then storage layout api calls are also logged as gcs calls.

Any error faced during fetching the bucket type is already logged at error level, and will show up in the logs separately . However, as for the GCS call log, the storage layout api returns with status OK even when error is faced and logged immediately afterwards due to the nature of implementation.

For e.g:

{"timestamp":{"seconds":1729840243,"nanos":34938039},"severity":"TRACE","message":"gcs: Req              0x0: <- GetStorageLayout(\"non-existent-bucket\")"}
{"timestamp":{"seconds":1729840243,"nanos":653348262},"severity":"ERROR","message":"Error returned from GetStorageLayout: rpc error: code = NotFound desc = The specified bucket does not exist."}
{"timestamp":{"seconds":1729840243,"nanos":653398322},"severity":"TRACE","message":"gcs: Req              0x0: -> GetStorageLayout(\"non-existent-bucket\") (618.464263ms): OK"}

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - Yes
  2. Unit tests - Yes
  3. Integration tests - Yes

@kislaykishore kislaykishore requested review from a team and BrennaEpp and removed request for a team October 25, 2024 06:46
@Tulsishah Tulsishah added the execute-integration-tests Run only integration tests label Oct 25, 2024
Copy link

codecov bot commented Oct 25, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.61%. Comparing base (5d1b6c5) to head (3d2ab47).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2629      +/-   ##
==========================================
- Coverage   77.63%   77.61%   -0.02%     
==========================================
  Files         109      109              
  Lines       15540    15550      +10     
==========================================
+ Hits        12064    12069       +5     
- Misses       2964     2968       +4     
- Partials      512      513       +1     
Flag Coverage Δ
unittests 77.61% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@anushka567 anushka567 force-pushed the add-gcs-call-log-for-storage-api branch from 5d840e9 to 48341f0 Compare October 25, 2024 07:39
@anushka567 anushka567 force-pushed the add-gcs-call-log-for-storage-api branch from 48341f0 to 3d2ab47 Compare October 25, 2024 07:41
@Tulsishah Tulsishah marked this pull request as ready for review October 25, 2024 07:49
@Tulsishah Tulsishah requested a review from a team as a code owner October 25, 2024 07:49
var err error

// BucketType internally calls GetStorageLayout API from control client , hence
// logging this as a GetStorageLayout call.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the call be logged even if GetStorageLayout GCS call has not happened ? BucketType() only calls the api once and uses the type throughout the lifecycle of mount

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Even when the actual network call has not happened, this is logged as a GCS call (time of request completion is considerably less).
Another scenario I am unsure about is when the first call comes in for a non hierarchical bucket with nil control client,

if bh.bucketType == gcs.Nil {
		if bh.controlClient == nilControlClient {
			bh.bucketType = gcs.NonHierarchical
			return bh.bucketType
		}

then GetStorageLayout api is not called, but this is also logged with minimal time taken for serving request.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then the logging information is not correct, the log should only come when gcs call happens

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
execute-integration-tests Run only integration tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants