docs: add performance advise for dynamic configurations

src/docs/asciidoc/README.adoc

@@ -7,7 +7,7 @@ AVIO Consulting
 :doctype: book
 :icons: font
 :toc: left
-:toclevels: 3
+:toclevels: 4
 :imagesdir: ./Images
 :tabsize: 4
 :keywords: Mule, MuleSoft, Observability, OpenTelemetry, OTel, Tracing, Instrumentation, Distributed

src/docs/asciidoc/module-config.adoc

@@ -463,29 +463,56 @@ image::manual-context-flow-injection.png[]
 
 [#_http_request_context_injection]
 ====== HTTP Request Context Injection
-The Module does **NOT** support automatic context propagation. In order to *propagate the trace header* information to other web applications, the Mule HTTP Requester Configuration *must*
-have default headers configured in the following way:
+The Module does **NOT** support automatic context propagation. In order to *propagate the trace header* information to other web applications, the Mule HTTP Requester Operation _OR_ Configuration *must* be configured to propagate following http headers -
 
-image::http-requester-config.png[600,600,title="Mule HTTP Requester Configuration",align="center"]
-
-
-.HTTP Requester Configuration for Default Headers
+.HTTP Headers for Trace Context
 [cols="30%, 70%"]
 |===
 | *Key*          | *Value*
 | `traceparent`  | `#[vars.OTEL_TRACE_CONTEXT.traceparent default '' as String]`
 | `tracestate`   | `#[vars.OTEL_TRACE_CONTEXT.tracestate default ''  as String]`
 |===
 
+*Using Operation vs. Global Configuration:*
+
+HTTP Outbound headers can be added at HTTP Request Operation level OR as a default headers in HTTP Request Global Configuration.
+
+HTTP Global configuration may be easier from code modifications perspective i.e. just at one global level. That comes at the cost of performance because using expressions in configurations makes it a *https://docs.mulesoft.com/mule-sdk/latest/static-dynamic-configs[dynamic configuration^]*.
+
+On the other hand, using headers section in HTTP Request operation may need modification of every `http:request` operation, but then it keeps the global configuration as a *static configuration* for runtime to reuse the same instance.
+
+WARNING: Using Static configurations is a preferred approach from performance perspective. Dynamic is only recommended for lower request volume uses cases. For more information on performance impact, see link:#_static_vs_dynamic_global_configurations[performance report section].
+
+*Propagation with Request Operation:*
+To propagate context using request operation, add/append request headers section of every `http:request` operation to include following entries -
+
+[source,xml]
+.Example GET Request with OTEL Trace context headers
+----
+<http:request method="GET" doc:name="Request" doc:id="f187cd33-70ce-4e09-96e0-70115c3ac727" config-ref="HTTP_Request_configuration" path="/api/path">
+    <http:headers ><![CDATA[#[output application/java
+---
+{
+"traceparent": vars.OTEL_TRACE_CONTEXT.traceparent as String,
+"tracestate": vars.OTEL_TRACE_CONTEXT.tracestate default '' as String
+}]]]>
+    </http:headers>
+</http:request>
+----
+
+*Propagation with Requester Global Configuration:*
+
+In order to propagate the trace header information to other web applications, the Mule HTTP Requester Configuration must have default headers configured in the following way:
 
 .Mule configuration xml for setting default headers in the HTTP Requester Configuration
 [source%nowrap%linenums, xml]
 ----
-<http:request-config name="HTTP_Request_configuration" doc:name="HTTP Request configuration" doc:id="7c863500-0642-4e9d-b759-5e317225e015" sendCorrelationId="NEVER">
-	<http:request-connection host="mule-hello-world-api.us-e1.cloudhub.io" />
+<http:request-config name="HTTP_Request_configuration"
+    doc:name="HTTP Request configuration">
+	<http:request-connection host="some-api.us-e1.cloudhub.io" />
 	<http:default-headers >
-		<http:default-header key='traceparent' value="#[vars.OTEL_TRACE_CONTEXT.traceparent default '' as String]" /> <1>
-		<http:default-header key='tracestate' value="#[vars.OTEL_TRACE_CONTEXT.tracestate default '' as String]" />   <2>
+		<http:default-header key='traceparent' value="#[vars.OTEL_TRACE_CONTEXT.traceparent default '' as String]" />
+		<http:default-header key='tracestate' value="#[vars.OTEL_TRACE_CONTEXT.tracestate default '' as String]" />
 	</http:default-headers>
 </http:request-config>
 ----

src/docs/asciidoc/performance.adoc

@@ -45,4 +45,59 @@ After multiple iterations of the test with above setup and comparing the perform
 * Up to 5% overhead on request processing under the load
 * Up to 10% CPU overhead
 
+[#_static_vs_dynamic_global_configurations]
+==== Static vs. Dynamic Configurations
 
+Depending on how global configurations are written they can be https://docs.mulesoft.com/mule-sdk/latest/static-dynamic-configs[Static or Dynamic].
+
+- *Static Configuration* - A configuration is static when none of its parameters are assigned an expression (even if those parameters do support expressions)
+
+[source,xml]
+----
+<http:request-config name="HTTP_Request_configuration" doc:name="HTTP Request configuration">
+    <http:request-connection host="${sys.greet.http.host}" port="${sys.greet.http.port}" />
+</http:request-config>
+----
+
+
+- *Dynamic Configuration* - In dynamic configurations, at least one parameter has an expression
+
+[source,xml]
+----
+<http:request-config name="HTTP_Request_configuration" doc:name="HTTP Request configuration">
+  <http:request-connection host="${sys.greet.http.host}" port="${sys.greet.http.port}"
+    <http:default-headers>
+      <http:default-header key="traceparent"
+value="#[vars.OTEL_TRACE_CONTEXT.traceparent as String]" />
+    </http:default-headers>
+</http:request-config>
+----
+
+_For Static configurations_, a single instance is created and _re-used_ each time it is needed.
+
+_For Dynamic configurations_, Each time an operation that points to a dynamic configuration is executed, all the parameters in the configuration are evaluated. A configuration instance is created for each unique set of resolved parameters. In that way, each time the same values are resolved, the same instance is returned, but for each different set of values, a _different_ instance is created.
+
+This lifecycle difference can make a huge difference on the configuration objects created. Every request will have a different `traceparent` value. Thus, when using Dynamic configurations, a new instance will be created *for each request*.
+
+Following result demonstrates the configuration Object creations for a load of _20 concurrent users for a 1-minute_ time period -
+
+*Observations:*
+
+- The number of Objects creation and allocation is much higher for Dynamic configurations
+- With configuration expiration window, this affects the memory consumption and GC times
+
+*Recommendations:*
+
+For higher load applications, static configurations i.e. propagation from operations instead of global configuration should be preferred.
+
+
+
+*Static Configuration Object Creations*
+
+image::Images/static_vs_dynamic_static_1.png[width=1024]
+image::Images/static_vs_dynamic_static_obj.png[width=1024]
+
+*Dynamic Configuration Object Creations*
+
+image::Images/static_vs_dynamic_dynamic_1.png[width=1024]
+image::Images/static_vs_dynamic_dynamic_obj.png[width=1024]