feat(metrics): introduce Metrics.flushMetrics (#2154)

* feat(metrics): introduce Metrics.flushMetrics

- introduce `Metrics.flushMetrics` as a more powerful version of `flushSingleMetrics` to allow
  - using defaults by inheriting state e.g. namespace, dimensions and metadata
  - emitting multiple metrics in one metrics context
- refactor `flushSingleMetrics` to use `flushMetrics`
- move namespace/service setting from `MetricsFactory` to `EmfMetricsLogger`

* feat(metrics): introduce Metrics.flushMetrics

- introduce `Metrics.flushMetrics` as a more powerful version of `flushSingleMetrics` to allow
  - using defaults by inheriting state e.g. namespace, dimensions and metadata
  - emitting multiple metrics in one metrics context
- refactor `flushSingleMetrics` to use `flushMetrics`
- move namespace/service setting from `MetricsFactory` to `EmfMetricsLogger`

* address metrics context issue

* fix addDimension wrongly adding to defaultDimensions

* introduce Metrics.addProperty and use it instead of addMetadata

* use flushMetrics in captureColdStartMetric

* use putProperty for addMetadata and consolidate duplicate logic

* update docs and javadoc

* Update docs/core/metrics.md

Co-authored-by: Philipp Page <philipp.page@yahoo.de>

---------

Co-authored-by: Philipp Page <philipp.page@yahoo.de>

Author: humanzz

docs/core/metrics.md

@@ -462,9 +462,9 @@ If you wish to set custom default dimensions, it can be done via `#!java metrics
     Overwriting the default dimensions will also overwrite the default `Service` dimension. If you wish to keep `Service` in your default dimensions, you need to add it manually.
 <!-- prettier-ignore-end -->
 
-### Creating a single metric with different configuration
+### Creating metrics with different configuration
 
-You can create a single metric with its own namespace and dimensions using `flushSingleMetric`:
+You can create metrics with different configurations e.g. different namespace and/or dimensions using `flushMetrics()`:
 
 === "App.java"
 
@@ -480,13 +480,17 @@ You can create a single metric with its own namespace and dimensions using `flus
         @Override
         @FlushMetrics(namespace = "ServerlessAirline", service = "payment")
         public Object handleRequest(Object input, Context context) {
-            metrics.flushSingleMetric(
-                "CustomMetric",
-                1,
-                MetricUnit.COUNT,
-                "CustomNamespace",
-                DimensionSet.of("CustomDimension", "value")  // Dimensions are optional
-            );
+            metrics.flushMetrics((customMetrics) -> {
+                customMetrics.addMetric("CustomMetric", 1, MetricUnit.COUNT);
+                // To optionally set a different namespace
+                customMetrics.setNamespace("CustomNamespace");
+                // To optionally set different default dimensions
+                customMetrics.setDefaultDimensions(DimensionSet.of("CustomDefaultDimension", "value"));
+                // To optionally append additional dimensions to default dimensions
+                customMetrics.addDimension(DimensionSet.of("CustomDimension", "value"));
+                // To optionally add metadata
+                customMetrics.addMetadata("CustomMetadata", "value"));
+            });
         }
     }
     ```
@@ -516,7 +520,7 @@ The following example shows how to configure a custom `Metrics` Singleton using
 
     public class App implements RequestHandler<Object, Object> {
         // Create and configure a Metrics singleton without annotation
-        private static final Metrics customMetrics = MetricsBuilder.builder()
+        private static final Metrics metrics = MetricsBuilder.builder()
             .withNamespace("ServerlessAirline")
             .withRaiseOnEmptyMetrics(true)
             .withService("payment")
@@ -527,11 +531,11 @@ The following example shows how to configure a custom `Metrics` Singleton using
             // You can manually capture the cold start metric
             // Lambda context is an optional argument if not available in your environment
             // Dimensions are also optional.
-            customMetrics.captureColdStartMetric(context, DimensionSet.of("FunctionName", "MyFunction", "Service", "payment"));
+            metrics.captureColdStartMetric(context, DimensionSet.of("FunctionName", "MyFunction", "Service", "payment"));
 
             // Add metrics to the custom metrics singleton
-            customMetrics.addMetric("CustomMetric", 1, MetricUnit.COUNT);
-            customMetrics.flush();
+            metrics.addMetric("CustomMetric", 1, MetricUnit.COUNT);
+            metrics.flush();
         }
     }
     ```

powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/Metrics.java

@@ -16,6 +16,7 @@
 
 import com.amazonaws.services.lambda.runtime.Context;
 import java.time.Instant;
+import java.util.function.Consumer;
 
 import software.amazon.lambda.powertools.metrics.model.DimensionSet;
 import software.amazon.lambda.powertools.metrics.model.MetricResolution;
@@ -162,7 +163,15 @@ default void captureColdStartMetric() {
     }
 
     /**
-     * Flush a single metric with custom dimensions. This creates a separate metrics context
+     * Flush a separate metrics context that inherits the namespace, default dimensions, and metadata. This creates a separate metrics context
+     * that doesn't affect the default metrics context.
+     *
+     * @param metricsConsumer the consumer to use to edit the metrics instance (e.g. add metrics, override namespace, set or add custom dimensions) before flushing
+     */
+    void flushMetrics(Consumer<Metrics> metricsConsumer);
+
+    /**
+     * Flush a single metric with custom namespace and dimensions. This creates a separate metrics context
      * that doesn't affect the default metrics context.
      *
      * @param name       the name of the metric
@@ -171,10 +180,17 @@ default void captureColdStartMetric() {
      * @param namespace  the namespace for the metric
      * @param dimensions custom dimensions for this metric (optional)
      */
-    void flushSingleMetric(String name, double value, MetricUnit unit, String namespace, DimensionSet dimensions);
+    default void flushSingleMetric(String name, double value, MetricUnit unit, String namespace, DimensionSet dimensions) {
+        flushMetrics(metrics -> {
+            metrics.setNamespace(namespace);
+            metrics.setDefaultDimensions(dimensions);
+            metrics.addMetric(name, value, unit);
+        });
+
+    }
 
     /**
-     * Flush a single metric with custom dimensions. This creates a separate metrics context
+     * Flush a single metric with custom namespace. This creates a separate metrics context
      * that doesn't affect the default metrics context.
      *
      * @param name       the name of the metric
@@ -183,6 +199,9 @@ default void captureColdStartMetric() {
      * @param namespace  the namespace for the metric
      */
     default void flushSingleMetric(String name, double value, MetricUnit unit, String namespace) {
-        flushSingleMetric(name, value, unit, namespace, null);
+        flushMetrics(metrics -> {
+            metrics.setNamespace(namespace);
+            metrics.addMetric(name, value, unit);
+        });
     }
 }

powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/EmfMetricsLogger.java

@@ -14,14 +14,14 @@
 
 package software.amazon.lambda.powertools.metrics.internal;
 
-import static software.amazon.lambda.powertools.common.internal.LambdaHandlerProcessor.getXrayTraceId;
 import static software.amazon.lambda.powertools.common.internal.LambdaHandlerProcessor.isColdStart;
 
 import java.time.Instant;
 import java.util.HashMap;
 import java.util.LinkedHashMap;
 import java.util.Map;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -43,16 +43,15 @@
  */
 public class EmfMetricsLogger implements Metrics {
     private static final Logger LOGGER = LoggerFactory.getLogger(EmfMetricsLogger.class);
-    private static final String TRACE_ID_PROPERTY = "xray_trace_id";
-    private static final String REQUEST_ID_PROPERTY = "function_request_id";
     private static final String COLD_START_METRIC = "ColdStart";
     private static final String METRICS_DISABLED_ENV_VAR = "POWERTOOLS_METRICS_DISABLED";
 
     private final software.amazon.cloudwatchlogs.emf.logger.MetricsLogger emfLogger;
     private final EnvironmentProvider environmentProvider;
-    private AtomicBoolean raiseOnEmptyMetrics = new AtomicBoolean(false);
+    private final AtomicBoolean raiseOnEmptyMetrics = new AtomicBoolean(false);
     private String namespace;
     private Map<String, String> defaultDimensions = new HashMap<>();
+    private final Map<String, Object> properties = new HashMap<>();
     private final AtomicBoolean hasMetrics = new AtomicBoolean(false);
 
     public EmfMetricsLogger(EnvironmentProvider environmentProvider, MetricsContext metricsContext) {
@@ -79,8 +78,6 @@ public void addDimension(software.amazon.lambda.powertools.metrics.model.Dimensi
         dimensionSet.getDimensions().forEach((key, val) -> {
             try {
                 emfDimensionSet.addDimension(key, val);
-                // Update our local copy of default dimensions
-                defaultDimensions.put(key, val);
             } catch (Exception e) {
                 // Ignore dimension errors
             }
@@ -91,7 +88,8 @@ public void addDimension(software.amazon.lambda.powertools.metrics.model.Dimensi
 
     @Override
     public void addMetadata(String key, Object value) {
-        emfLogger.putMetadata(key, value);
+        emfLogger.putProperty(key, value);
+        properties.put(key, value);
     }
 
     @Override
@@ -173,45 +171,13 @@ public void flush() {
     public void captureColdStartMetric(Context context,
             software.amazon.lambda.powertools.metrics.model.DimensionSet dimensions) {
         if (isColdStart()) {
-            if (isMetricsDisabled()) {
-                LOGGER.debug("Metrics are disabled, skipping cold start metric capture");
-                return;
-            }
-
-            Validator.validateNamespace(namespace);
-
-            software.amazon.cloudwatchlogs.emf.logger.MetricsLogger coldStartLogger = new software.amazon.cloudwatchlogs.emf.logger.MetricsLogger();
-
-            try {
-                coldStartLogger.setNamespace(namespace);
-            } catch (Exception e) {
-                LOGGER.error("Namespace cannot be set for cold start metrics due to an error in EMF", e);
-            }
-
-            coldStartLogger.putMetric(COLD_START_METRIC, 1, Unit.COUNT);
-
-            // Set dimensions if provided
-            if (dimensions != null) {
-                DimensionSet emfDimensionSet = new DimensionSet();
-                dimensions.getDimensions().forEach((key, val) -> {
-                    try {
-                        emfDimensionSet.addDimension(key, val);
-                    } catch (Exception e) {
-                        // Ignore dimension errors
-                    }
-                });
-                coldStartLogger.setDimensions(emfDimensionSet);
-            }
-
-            // Add request ID from context if available
-            if (context != null && context.getAwsRequestId() != null) {
-                coldStartLogger.putProperty(REQUEST_ID_PROPERTY, context.getAwsRequestId());
-            }
-
-            // Add trace ID using the standard logic
-            getXrayTraceId().ifPresent(traceId -> coldStartLogger.putProperty(TRACE_ID_PROPERTY, traceId));
-
-            coldStartLogger.flush();
+            flushMetrics(metrics -> {
+                MetricsUtils.addRequestIdAndXrayTraceIdIfAvailable(context, metrics);
+                if (dimensions != null) {
+                    metrics.setDefaultDimensions(dimensions);
+                }
+                metrics.addMetric(COLD_START_METRIC, 1, MetricUnit.COUNT);
+            });
         }
     }
 
@@ -221,43 +187,24 @@ public void captureColdStartMetric(software.amazon.lambda.powertools.metrics.mod
     }
 
     @Override
-    public void flushSingleMetric(String name, double value, MetricUnit unit, String namespace,
-            software.amazon.lambda.powertools.metrics.model.DimensionSet dimensions) {
+    public void flushMetrics(Consumer<Metrics> metricsConsumer) {
         if (isMetricsDisabled()) {
             LOGGER.debug("Metrics are disabled, skipping single metric flush");
             return;
         }
-
-        Validator.validateNamespace(namespace);
-
-        // Create a new logger for this single metric
-        software.amazon.cloudwatchlogs.emf.logger.MetricsLogger singleMetricLogger = new software.amazon.cloudwatchlogs.emf.logger.MetricsLogger(
-                environmentProvider);
-
-        try {
-            singleMetricLogger.setNamespace(namespace);
-        } catch (Exception e) {
-            LOGGER.error("Namespace cannot be set for single metric due to an error in EMF", e);
+        // Create a new instance, inheriting namespace, default dimensions, and metadata
+        EmfMetricsLogger metrics = new EmfMetricsLogger(environmentProvider, new MetricsContext());
+        if (namespace != null) {
+            metrics.setNamespace(this.namespace);
         }
-
-        // Add the metric
-        singleMetricLogger.putMetric(name, value, convertUnit(unit));
-
-        // Set dimensions if provided
-        if (dimensions != null) {
-            DimensionSet emfDimensionSet = new DimensionSet();
-            dimensions.getDimensions().forEach((key, val) -> {
-                try {
-                    emfDimensionSet.addDimension(key, val);
-                } catch (Exception e) {
-                    // Ignore dimension errors
-                }
-            });
-            singleMetricLogger.setDimensions(emfDimensionSet);
+        if (!defaultDimensions.isEmpty()) {
+            metrics.setDefaultDimensions(software.amazon.lambda.powertools.metrics.model.DimensionSet.of(defaultDimensions));
         }
+        properties.forEach(metrics::addMetadata);
+
+        metricsConsumer.accept(metrics);
 
-        // Flush the metric
-        singleMetricLogger.flush();
+        metrics.flush();
     }
 
     private boolean isMetricsDisabled() {

powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/LambdaMetricsAspect.java

@@ -34,8 +34,6 @@
 
 @Aspect
 public class LambdaMetricsAspect {
-    public static final String TRACE_ID_PROPERTY = "xray_trace_id";
-    public static final String REQUEST_ID_PROPERTY = "function_request_id";
     private static final String SERVICE_DIMENSION = "Service";
     private static final String FUNCTION_NAME_ENV_VAR = "POWERTOOLS_METRICS_FUNCTION_NAME";
 
@@ -90,11 +88,10 @@ public Object around(ProceedingJoinPoint pjp,
 
             metricsInstance.setRaiseOnEmptyMetrics(metrics.raiseOnEmptyMetrics());
 
-            // Add trace ID metadata if available
-            LambdaHandlerProcessor.getXrayTraceId()
-                    .ifPresent(traceId -> metricsInstance.addMetadata(TRACE_ID_PROPERTY, traceId));
+            Context extractedContext = extractContext(pjp);
+            MetricsUtils.addRequestIdAndXrayTraceIdIfAvailable(extractedContext, metricsInstance);
 
-            captureColdStartMetricIfEnabled(extractContext(pjp), metrics);
+            captureColdStartMetricIfEnabled(extractedContext, metrics);
 
             try {
                 return pjp.proceed(proceedArgs);
@@ -107,40 +104,36 @@ public Object around(ProceedingJoinPoint pjp,
         return pjp.proceed(proceedArgs);
     }
 
-    private void captureColdStartMetricIfEnabled(Context extractedContext, FlushMetrics metrics) {
+    private void captureColdStartMetricIfEnabled(Context extractedContext, FlushMetrics flushMetrics) {
         if (extractedContext == null) {
             return;
         }
 
-        Metrics metricsInstance = MetricsFactory.getMetricsInstance();
-        // This can be null e.g. during unit tests when mocking the Lambda context
-        if (extractedContext.getAwsRequestId() != null) {
-            metricsInstance.addMetadata(REQUEST_ID_PROPERTY, extractedContext.getAwsRequestId());
-        }
+        Metrics metrics = MetricsFactory.getMetricsInstance();
 
         // Only capture cold start metrics if enabled on annotation
-        if (metrics.captureColdStart()) {
+        if (flushMetrics.captureColdStart()) {
             // Get function name from annotation or context
-            String funcName = functionName(metrics, extractedContext);
+            String funcName = functionName(flushMetrics, extractedContext);
 
-            DimensionSet coldStartDimensions = new DimensionSet();
+            DimensionSet dimensionSet = new DimensionSet();
 
             // Get service name from metrics instance default dimensions or fallback
-            String serviceName = metricsInstance.getDefaultDimensions().getDimensions().getOrDefault(
+            String serviceName = metrics.getDefaultDimensions().getDimensions().getOrDefault(
                     SERVICE_DIMENSION,
-                    serviceNameWithFallback(metrics));
+                    serviceNameWithFallback(flushMetrics));
 
             // Only add service if it is not undefined
             if (!LambdaConstants.SERVICE_UNDEFINED.equals(serviceName)) {
-                coldStartDimensions.addDimension(SERVICE_DIMENSION, serviceName);
+                dimensionSet.addDimension(SERVICE_DIMENSION, serviceName);
             }
 
             // Add function name
             if (funcName != null) {
-                coldStartDimensions.addDimension("FunctionName", funcName);
+                dimensionSet.addDimension("FunctionName", funcName);
             }
 
-            metricsInstance.captureColdStartMetric(extractedContext, coldStartDimensions);
+            metrics.captureColdStartMetric(extractedContext, dimensionSet);
         }
     }
 }

powertools-metrics/src/main/java/software/amazon/lambda/powertools/metrics/internal/MetricsUtils.java

@@ -0,0 +1,22 @@
+package software.amazon.lambda.powertools.metrics.internal;
+
+import com.amazonaws.services.lambda.runtime.Context;
+import software.amazon.lambda.powertools.metrics.Metrics;
+
+import static software.amazon.lambda.powertools.common.internal.LambdaHandlerProcessor.getXrayTraceId;
+
+final class MetricsUtils {
+    private static final String TRACE_ID_PROPERTY = "xray_trace_id";
+    private static final String REQUEST_ID_PROPERTY = "function_request_id";
+
+    private MetricsUtils() {
+        // Utility class
+    }
+
+    static void addRequestIdAndXrayTraceIdIfAvailable(Context context, Metrics metrics) {
+        if (context != null && context.getAwsRequestId() != null) {
+            metrics.addMetadata(REQUEST_ID_PROPERTY, context.getAwsRequestId());
+        }
+        getXrayTraceId().ifPresent(traceId -> metrics.addMetadata(TRACE_ID_PROPERTY, traceId));
+    }
+}