Forward deployment configuration errors to asker from Google Cloud Run

Contents

Enhancements

• Forward deployment configuration errors to asker from Cloud Run (errors raised in the analysis are already forwarded)

Operations

• Add enhancement branch prefix type to pre-commit config

Ensure datafile metadata can be set at instantiation for bare cloud objects

Contents

Fixes

• Allow local metadata values to override cloud values if provided at Datafile instantiation

Use latest conventional commits package for CI/CD workflows

Contents

Operations

• Use new pull request description workflow
• Use latest conventional commits package and update workflow

Stream responding service logs to asker via pub/sub

Summary

Add the option to stream the responding service logs to the asker via Google Cloud Pub/Sub. The asker then adds [REMOTE] to the start of the log messages and handles them with its own log handlers. The scientist can turn this option on or off each time they ask one of their analysis's children a question.

Contents

Enhancements

• Add GooglePubSubHandler, which publishes log records to a Google Pub/Sub topic
• Add option to send log messages created during the running of an analysis in Service.answer to the answer topic
• Allow asker to receive and handle any number of non-final-response messages before it deletes the answer topic and subscription
• Allow easy modular addition of new message handlers
• Make the asking service handle log messages forwarded to it by the responding service
• Ensure the asker handles pub/sub messages in the order they were sent by the responder, not the order they were received - this ensures remote log messages are always printed in order and before the final result is logged as received
• Get asker to log receipt of messages of unknown types
• Improve Service log messages
• Always send analysis log messages to stderr as well as optionally sending them to the given log handler in Runner.run

Fixes

• Apply formatter to any handler passed to apply_log_handler
• Fix local log notification of logs being streamed to a remote URI
• Add USE_OCTUE_LOG_HANDLER environment variable to opt-in to octue log handler (default value opts out) so that importing octue doesn't add a potentially unwanted log handler

Refactoring

• Rename logging_handers module to log_handlers
• Move log_level and log_handler parameters from Runner.__init__ to Runner.run and update usages
• Factor out pulling of messages from a subscription in Service
• Remove thread and process information from logging context

Operations

• Fix mkver.conf so new features trigger a minor version increase

Documentation

• Update and condense some docstrings

Quality Checklist

• New features are fully tested (No matter how much Coverage Karma you have)

Add short version of equals filter and add missing filters

Contents

Enhancements

• Add shortcut for equals filter

Fixes

• Ensure all defined types of a Filterable's attributes have equals and is filters
• Treat KeyErrors and TypeErrors equally in getattr_or_subscribe

Allow new datafiles to be created in the cloud without a local file existing first

Contents

Enhancements

• Allow new datafiles to be created in cloud without local file

Fix log handling

Contents

Enhancements

• Improve Service log messages
• Keep full module path in logging context when using analysis logger
• Demote storage client log messages to DEBUG level

Fixes

• Fix and improve apply_log_handler function
• Apply log handler to the root logger in top-level __init__ module
• Stop duplication of timestamp in Google Cloud Run logs by applying different handler/formatter if PLATFORM environment variable is present and equal to GOOGLE_CLOUD_RUN
• Add COMPUTE_PLATFORM=GOOGLE_CLOUD_RUN environment variable to docker image
• Remove applying of log handler from incorrect places
• Remove extra logger in runner module
• Stop analysis logger propagating to the root logger and repeating log messages

Reversions

• Remove show-twined-logs option from CLI

Testing

• Silence GCP Storage emulator logs below ERROR level
• Ensure unmocked Subscription test always deletes topic and subscription
• Remove extra logger from tests.base
• Avoid irrelevant exception in Cloud Run tests

Operations

• Remove unnecessary preceding underscores from ARGs in Dockerfile

Use question UUID as analysis ID

Summary

Use service question UUIDs as analysis IDs to allow log messages for a Google Cloud Run service to be filtered for a given question. Also improve the presentation of error messages forwarded from the responding service to the asking service.

Contents

Enhancements

• Use service question UUIDs as analysis IDs
• Dynamically create exception class in Service._raise_exception_from_responder if the received type name is unknown to the asker

Minor improvements

• Use a string UUID for question IDs rather than an integer UUID

Allow non-JSON-encoded items in GCS metadata

Contents

Fixes

• Allow non-JSON-encoded items in GCS metadata (this means a Datafile can be instantiated from any GCS object with arbitrary metadata rather than raising an error)

Remove `sequence` and `cluster` attributes from Datafile

Contents

Breaking changes

• Remove the sequence and cluster attributes from Datafile (they were simply tags with unnecessary special treatment)

Refactoring

• Replace Dataset.get_file_sequence with added functionality in FilterContainer.order_by

Operations

• Ensure breaking change commits increment minor version number in mkver.conf