Repository navigation
allow sending extras when setting up client #15423
Description
Activity
It would be great if trace (
X-Cloud-Trace-Contextheader) was included automatically. For example Cloud Run logging is super confusing. The documentation just points all around without leading to any clear solutions and Google-searches direct to some random GitHub repos with people having logging issues.So please consider adding tracing even in the simplest Cloud Run apps that are basically
import logging logging.info("message")Reacted by Ryan Barrett and Santiago Piqueras- added a commit that references this issue
on Mar 6, 2021 It would be great if trace (X-Cloud-Trace-Context header) was included automatically. For example Cloud Run logging is super confusing. The documentation just points all around without leading to any clear solutions and Google-searches direct to some random GitHub repos with people having logging issues.
So please consider adding tracing even in the simplest Cloud Run apps that are basically
import logging
logging.info("message")Please. It is mind-blowing how badly these libraries integrate with Logging Explorer. This two things:
- grouping logs by request automatically How to group logs by request like python 2.7 standard? python-logging#587
- parent log inheriting the highest child severity Inherit log severity on parent log entry from highest level child entry python-logging#91
have been requested multiple times, and were already a thing in all Gen1 App Engine environments, yet people have to resort to third party solutions (such as https://github.com/salrashid123/flask-gcp-log-groups) to make the logs bearable. I can't think of any use case where you wouldn't want those things to just happen.
Reacted by Johnny Kutnowski, Gianfranco P and Alfonso Montero LópezHey @spiqueras ,
It's surprising to me that you're still seeing these issues. The library does attempt to parse X-Cloud-Trace-Context and w3 traceparent headers for trace correlation. The Log grouping backend behaviour has changed since App Engine Gen1, but it should now group based on
http_requestmetadata. The library should attempt to populate these http fields automatically for Flask and django, which should result in log grouping (although maybe not exactly like the old App Engine Gen 1 method).It's possible the issues you're seeing are due to a bug in the library, or it could be a configuration error, or it could just be an issue with the environment you are running the code in. If you can give me more information of how you're running your code, example logs, and/or sample code that reproduces what you're seeing, I can try to take a look.
If you have an example of a log output by the library that's ungrouped vs a similar one that is grouped how you expect, that would be especially helpful in tracking down the root cause
Thanks,
DanielFor anyone using this library who is only using it because they want to get log correlation like they had before with appengine standard, and who doesn't need to actually call functions in the cloud logging API, you don't need to use this library at all. You might have to change the way you are looking at logs, or in the way you are analyzing their output if you are forwarding them to BigQuery or elsewhere using log sinks and depend on
protoPayload.line.logMessageetc - that seems to be just the way it is.
Please check here and here - this was all we needed.
The example in the first link is for cloud run but also works for appengine. It just shows you how to do this usingprint()- which I guess you could do, but really I'm not sure why one would want to. You can create a subclass ofLogFormatterthat returns the same json-dumped string as described there from itsformatmethod, set it as the formatter of the root logger, and you'll have correlated logs without using this library at all. You can havelogging.info("Hi, this is a plaintext string")and have all app logs such as this properly grouped per request, including the usual metadata you'd expect such asseverity(see the second link for how to do that), without importing this package.I'm not sure why they don't recommend this approach by default for appengine customers in this situation, and maybe there's a good reason I'm not aware of, so take this comment with a grain of salt. I'm sharing this here because it would have saved me many hours of wasted time if I had seen a comment like this.
Hey pnico,
Yes, you are correct that App Engine and certain other GCP environments will automatically pick up formatted stdout logs. This library provides the StructuredLogHandler to take advantage of this, which essentially does what you describe, by sending logs through a custom Formatter and Handler to export properly formatted logs over stdout, to avoid the overhead of network calls.
This library will attempt to determine which environment your code is running on, and pick the best log export method based on the environment. But you are correct that if you only plan on running your code on App Engine, you can avoid this library and just write structured json logs directly (using our StructuredLogHandler, or custom code doing something similar)
Yeah, I looked at some of the handler classes there but importing any of those seems to bring in the rest of the library
python3 Python 3.11.2 (v3.11.2:878ead1ac1, Feb 7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import psutil >>> psutil.Process().memory_info().rss / (1024 * 1024) 12.63671875 >>> from google.cloud.logging_v2.handlers import StructuredLogHandler >>> psutil.Process().memory_info().rss / (1024 * 1024) 53.328125 >>>Maybe those handlers add some details that could be useful, but we can get everything we need with something much simpler. GCP is actually adding resource labels
module_id,project_id,version_idautomatically, even.This could be addressed with an adapter: #15414
@gkevinzheng @daniel-sanche fancy to provide at least a cursory review to the PR? (I'm not able due to my limited skills)
I guess many people are interested in the feature and ruling out showstoppers from the PR would allow others to take it for a spin and provide early feedback (as I do).The code in googleapis/python-logging#751 looks safe to use if you'd find it helpful, although it only adds labels to logs, not other fields like was initially discussed
I don't think this feature is likely to be added to python-logging in the near-term, but the library fully supports external adapters in user code
Transferring to google-cloud-python as the code for python-logging will be migrated there in 1-3 weeks.
Closing based on #15423 (comment)
We recently added a feature to allow log commands to send in
extras, allowing users to customize certain fields (googleapis/python-logging#129)@nicoleczhu suggested expanding this by allowing users to set default
extraswhen first setting up the logging client