RFC for Open Telemetry Implementation for Presto. #33

sureshbabu-areekara · 2024-11-08T04:30:14Z

No description provided.

tdcmeehan · 2024-11-08T15:19:32Z

RFC-0009-open-telemetry.md

+span-sampling=true
+```
+
+***otel-factory.name***: unique identifier for OpenTelemetry factory implementation to be registered


This seems to imply that the only tracing providers supported must implement the Open Telemetry spec. The runtime should have no knowledge of any aspect of the Open Telemetry spec. The Open Telemetry spec should be implemented as a plugin and the SPI integration with the runtime should have a generic concept of tracing without any knowledge of any particular spec.

As an example, the Presto runtime and execution engines do not have any knowledge of the Iceberg spec either. They communicate through the connector SPI.

Please update your proposal with specifics on the abstractions that will be used within the SPI to accomplish this richer tracing functionality. Thank you.

Hi @tdcmeehan The abstraction in SPI is generic and can be implemented by any tracing providers. If the key with "otel" and "OpenTelemetry factory" in the description are confusing here we could modify to some generic name like "tracing-factory-name" and "factory".

Please add details on the changes being made to the SPI in this document.

Hi @tdcmeehan Added the SPI details. Please review.

I don't see it. Can you add actual interfaces and update the flow chart to denote runtime/engine -> libray, with the arrow indicating which SPI method will be invoked?

Hi @tdcmeehan Added the flow chart. Could you please check. Thank you!!

prestodb-ci · 2024-11-19T02:36:03Z

Saved that user @sureshbabu-areekara is from IBM

sureshbabu-areekara · 2024-11-25T05:24:58Z

Hi @tdcmeehan Please find the draft PR prestodb/presto#24133

aaneja · 2025-03-03T09:22:14Z

RFC-0009-open-telemetry.md

@@ -0,0 +1,113 @@
+# **RFC0009 for Presto**


You will need to change this RFC0011 since 9 & 10 were already commited

aaneja · 2025-03-03T09:27:48Z

RFC-0009-open-telemetry.md

+`BaseSpan` - SPI can be extended with the specific implementation which contains the actual SDK Span. BaseSpan propagates through the main presto code.
+`Tracer` - SPI for the span operations which can be implemented by specific serviceability framework.
+`TraceProvider` - SPI to supply different `Tracer` implementations.

-As of now we have one implementation for Open Telemetry can be configured by modifying the values in presto-main/etc/telemetry.properties
+`BaseSpan` introduced as part of new design. Hooks inside `Tracer` and `TraceProvider` is added/updated to match the design.
+`NoopTracer` removed as we have a default `Tracer` implementation in `presto-main/..tracing/DefaultTelemetryTracer`
+
+`Tracer` implementation get loaded by `PluginManager` at startup as before.
+
+Trace can be configured by modifying the values in presto-main/etc/telemetry-tracing.properties


Line endings are incorrect here. You need to add double spaces to get a newline in the rendered markdown

Instead of pasting the various fields of the SPI, please paste the additions and deletions made to the SPI classes as code blocks

aaneja · 2025-03-03T09:29:57Z

RFC-0009-open-telemetry.md


-***exporter-timeout***: how long the span exporter will wait for a batch of spans to be successfully sent before timing out
+***trace-sampling-ratio***: Double between 0.0 and 1.0 to specify the percentage of queries to be traced.


How is sampling done ? Is there a way to control which queries get sampled ?

0.0 : No requests are traced.
1.0 : Every request is traced.
Values between 0.0 to 1.0 : A fraction of requests are randomly traced. So we can decide how much % of queries to be sampled to reduce overhead.

Custom Sampling can be implemented for a particular type of query by overriding shouldSample method in Sampler class. But currently this is not implemented.

aaneja · 2025-03-03T09:31:26Z

RFC-0009-open-telemetry.md

+Propagation is usually handled by instrumentation libraries and is transparent to the user. In the event that you need to manually propagate context, you can use the Propagators API.
+
+OpenTelemetry maintains several official propagators. The default propagator is using the headers specified by the W3C TraceContext specification.
+- In Presto in areas where REST calls involved, we use the header for context propagation as per the above image.


Can you describe the format of the header ? What are the key and value storing ? How big is the header ?

Format :
traceparent: {version}-{trace-id}-{parent-id}-{trace-flags}

Ex :
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Length :

55 bytes is the total length.

aaneja · 2025-03-03T09:32:57Z

RFC-0009-open-telemetry.md

+
+OpenTelemetry is a powerful serviceability framework that helps to gain insights into the performance and behaviour of the systems. It facilitates generation, collection, and management of telemetry data such as traces.
+
+The OSS Presto had a basic implementation of Open Telemetry.


Can you make a new section called Existing Open Telemetry implementation and call out

What flows are traced via the existing implementation

What flows you think are missing and should be traced

aaneja · 2025-03-03T09:34:05Z

RFC-0009-open-telemetry.md

+
+The OSS Presto had a basic implementation of Open Telemetry.
+
+![Traces existing implementation](/RFC-0009-open-telemetry/traces-existing-implementation-oss-presto.png)


This screenshot is not helpful. Instead let's use the list of flows traced (describe them in a few sentences) to motivate the problem

aaneja · 2025-03-03T09:35:09Z

RFC-0009-open-telemetry.md

+
+## Proposed Implementation
+
+Presto can be manually instrumented and will have the following advantages.


We already have an existing OpenTelemetry related SPI. Let's change this to -

Suggested change

Presto can be manually instrumented and will have the following advantages.

With additional changes to the OpenTelemetry SPI, we get the following advantages -

aaneja · 2025-03-03T09:36:31Z

RFC-0009-open-telemetry.md

+
+![Instrumentation flow](/RFC-0009-open-telemetry/tracing-instrumentation-flow.png)
+- Open Telemetry SDK provides libraries for instrumenting applications to capture telemetry data(traces). It includes built-in integrations for common frameworks and supports custom instrumentation.
+- Presto application is getting instrumented using OpenTelemetry API.


Remove this line, adds no value

aaneja · 2025-03-03T09:38:52Z

RFC-0009-open-telemetry.md

+- Ability to pass additional information as span attributes and events
+
+![Instrumentation flow](/RFC-0009-open-telemetry/tracing-instrumentation-flow.png)
+- Open Telemetry SDK provides libraries for instrumenting applications to capture telemetry data(traces). It includes built-in integrations for common frameworks and supports custom instrumentation.


Looks like you're describing the interaction diagram from the image. Can you add a subsection heading like ### Interactions in the new flow ?
Instead of describing the full interaction (which the diagram is covering), only add details that are missing or not covered in the diagram

aaneja · 2025-03-03T09:39:32Z

RFC-0009-open-telemetry.md

+- The SDK continuously checks the flush trigger, and when it reaches the batch size, all spans are batched and sent to the backend.
+- Backend is a system to store, analyse and visualize this telemetry data. Common backends include systems like Jaeger, Instana, Grafana stack, etc.
+
+![Context propagation](/RFC-0009-open-telemetry/context-propagation-coordinator-to-worker.png)


Please use sub-headings to describe what you're covering.

aaneja · 2025-03-03T09:40:14Z

RFC-0009-open-telemetry.md

@@ -67,9 +67,16 @@ OpenTelemetry maintains several official propagators. The default propagator is
 Based on the discussion, this may need to be updated with feedback from reviewers.

 ## Adoption Plan


Please add a sub-heading called SPI Changes

ZacBlanco · 2025-03-04T19:17:27Z

RFC-0011-open-telemetry.md

+
+
+## Summary
+The existing Open Telemetry implementation https://github.com/prestodb/presto/pull/18534 was an experimental feature, had a limited set of telemetry data(Query state changes) and did not include a child span concept. The recent implementation will make Presto more flexible, allowing support for both parent and child spans. Additionally, traces can now be propagated to the worker nodes as well.


This should summarize what the document is proposing, there can be a sample implementation, but the design is more important than implementation in the RFC. We need to highlight the shortcomings of the current tracing provider SPI and propose changes to the current SPI to support the desired features.

It seems like the desired features are:

child spans

tracing propagated to workers

In the later sections we need to explain why the current interface does not allow these two features.

If there are more features required, they should be called out explicitly

Suggested change

The existing Open Telemetry implementation https://github.com/prestodb/presto/pull/18534 was an experimental feature, had a limited set of telemetry data(Query state changes) and did not include a child span concept. The recent implementation will make Presto more flexible, allowing support for both parent and child spans. Additionally, traces can now be propagated to the worker nodes as well.

Presto's original tracing implementation was merged in https://github.com/prestodb/presto/pull/18534. It was an experimental feature, had a limited set of telemetry data, and did not include a child span concept. This document proposes an expansion to the tracing provider SPI to

1. Support child spans in traces to make understanding nested operations simpler

2. Allow traces to be propagated to workers so that the entire lifespan of a query can be traced from analysis through execution

ZacBlanco · 2025-03-04T19:21:13Z

RFC-0011-open-telemetry.md

+
+
+## Background
+OpenTelemetry is a powerful serviceability framework that helps to gain insights into the performance and behaviour of the systems. It facilitates generation, collection, and management of telemetry data such as traces.


We should drop the term "Open Telemetry" as much as possible from the document because we're trying to design an SPI interface that open telemetry is capable of implementing. The SPI tracing provider interface is the important thing to highlight.

ZacBlanco · 2025-03-04T19:23:31Z

RFC-0011-open-telemetry.md

+OpenTelemetry is a powerful serviceability framework that helps to gain insights into the performance and behaviour of the systems. It facilitates generation, collection, and management of telemetry data such as traces.
+
+
+### Existing Open Telemetry


We should update this to describe the current TracerProvider/Tracer interfaces and specifically describe why they are insufficient

ZacBlanco · 2025-03-04T19:25:07Z

RFC-0011-open-telemetry.md

+![Traces existing implementation](/RFC-0011-open-telemetry/traces-existing-implementation-oss-presto.png)
+
+
+The OSS Presto had a basic implementation of Open Telemetry with few limitations listed below.


Suggested change

The OSS Presto had a basic implementation of Open Telemetry with few limitations listed below.

Presto has a basic `TracerProvider` interface which makes it difficult to implement the following desirable features:

ZacBlanco · 2025-03-04T19:25:37Z

RFC-0011-open-telemetry.md

+- No correlation between traceId and queryId.
+- Does not identify failed queries
+- No context propagation to track worker nodes.
+


Here you should point out specifically which SPI methods used for tracing make implementing the particular features difficult

ZacBlanco · 2025-03-04T19:30:58Z

RFC-0011-open-telemetry.md

+Runnable getCurrentContextWrap(Runnable runnable);
+boolean isRecording();
+Map<String, String> getHeadersMap(T span);
+void endSpanOnError(T querySpan, Throwable throwable);
+void addEvent(T span, String eventName);
+void setAttributes(T span, Map<String, String> attributes);
+void recordException(T span, String message, RuntimeException runtimeException, ErrorCode errorCode);
+void setSuccess(T span);
+T getInvalidSpan();
+T getRootSpan(String traceId);
+T getSpan(String spanName);
+U scopedSpan(String name, Boolean... skipSpan);
+Optional<String> spanString(T span);


These all should have javadoc comments explaining what they do. Ideally the RFC should also explain how and where they are used.

ZacBlanco · 2025-03-04T19:32:15Z

RFC-0011-open-telemetry.md

+#### Deletions
+```java
+String tracerName = "com.facebook.presto";
+void addPoint(String annotation);
+void startBlock(String blockName, String annotation);
+void addPointToBlock(String blockName, String annotation);
+void endBlock(String blockName, String annotation);
+void endTrace(String annotation);
+```


We cannot just delete from the SPI, so we must add to the transition plan how we will either (1) maintain compatibility with the existing tracing methods and (2) eventually transition to the new API entirely. Will there be a feature flag?

@ZacBlanco @aaneja As the existing OTel feature was and experimental one with limited spans, coexistence of both is relevant and a feasible approach?

ZacBlanco · 2025-03-04T19:32:39Z

RFC-0011-open-telemetry.md

+```java
+class TraceProvider {}
+```
+SPI to supply different Tracer implementations.


We cannot delete this because we must be able to supply multiple tracer implementations

TraceProvider completely not removed. only some methods in it.

ZacBlanco · 2025-03-04T19:37:08Z

RFC-0011-open-telemetry.md

+
+
+### Configuration
+Trace can be configured by modifying the values in presto-main/etc/telemetry-tracing.properties


This seems specific to the OpenTelemetry implementation. I think it's fine to propose changes to how we configure the existing OpenTelemtry tracing plugin, but we need to be very clear about the any configuration properties that would be added to TracingConfig since it affects config.properties

Thanks for the feedback! We'll make sure to clearly document it. Also Added the current configuration along with proposed configuration in RFC.

ZacBlanco · 2025-03-04T19:39:54Z

RFC-0011-open-telemetry.md

+## Proposed Implementation
+With additional changes to the OpenTelemetry SPI, we get the following advantages -
+- Presto can be manually instrumented which gives more flexibility and control. Easier to customize what operations can be monitored.
+- More number of spans with parent and child span concept which helps to get the insight on internal operations.
+- Able to pass queryId as attribute so that traces and query get connected each other.
+- Can identify failed queries and filter it out.
+- Ability to pass additional information as span attributes and events.
+- Context propagation from coordinator to worker nodes.
+
+


This section needs to spell out exactly what changes are being made to the TracerProvider and Tracer interfaces in java code, and also any additional architectural changes, such as things that the C++ and Java workers might have to implement. The diagrams are good, but we need to understand the exact changes to the existing interfaces. It isn't clear what the changes are just looking at the diagrams.

I see there is an SPI section below, but it needs to be moved up here and the exact changes to the flow of tracing spans/blocks must be described in text. The descriptions also need to spell out why the existing Tracer/TracerProvider interfaces are insufficient and how the new SPI changes make the desired features possible

Hi @ZacBlanco @aaneja Moved SPI changes as per the comment #33 (comment) . Please let me know where to keep. Under Adoption Plan or Proposed implementation ?

Updated what all additions in instrumentation flow with the new proposal. Context propagation from Coordinator to Worker is entirely new for the proposed implementation.

ZacBlanco · 2025-03-07T23:12:08Z

RFC-0011-open-telemetry.md

+scheduling. ie. it does not cover most of the internal operations during a query execution(The implementation for methods startBlock and endBlock are based on Map and cannot be used to handle large number of spans as the span name can be same for many.).
+- No additional information is being passed as query attributes or events. No suitable methods in SPI to set events and attributes.
+- No correlation between traceId and queryId. No suitable methods in SPI to set queryId.
+- Does not identify failed queries. No suitable methods in SPI to pass Error/Exception information and deals with failed query spans.


Suggested change

- Does not identify failed queries. No suitable methods in SPI to pass Error/Exception information and deals with failed query spans.

- Does not identify failed queries. No suitable methods in SPI to pass Error/Exception information and details with failed query spans.

Is this supposed to be "details"?

ZacBlanco · 2025-03-07T23:16:00Z

RFC-0011-open-telemetry.md

+SPI can be extended with the specific implementation which contains the actual SDK Span. BaseSpan propagates through the main presto code. Introduced as part of new design.
+
+```java
+  default void close()


are these methods on the class of BaseSpan? It is not clear. Also, default is not valid unless BaseSpan is defined as an interface.

How do the methods close() and end() differ? When should they be used?

BaseSpan is an SPI interface which is autoclosable. close() came from AutoCloseable and end() part of BaseSpan.
close() can be used to end span in a try-with-resource and end() to manually end the span.

ZacBlanco · 2025-03-07T23:18:01Z

RFC-0011-open-telemetry.md

+```java
+class Tracer {}
+```
+SPI for the span operations which can be implemented by specific serviceability framework.


If this is meant to be a comment on the Tracer class, can you add it as a javadoc above the class within the code block?

ZacBlanco · 2025-03-07T23:18:43Z

RFC-0011-open-telemetry.md

+```java
+class TraceProvider {}
+```
+SPI to supply different Tracer implementations.


Similar comment on moving this to a javadoc.

ZacBlanco · 2025-03-07T23:19:04Z

RFC-0011-open-telemetry.md

+```java
+  T create();
+```


If this belongs on TracerProvider please put it in the same code block

ZacBlanco · 2025-03-07T23:32:25Z

RFC-0011-open-telemetry.md

+   * @param runtimeException the runtime exception
+   * @param errorCode        the error code
+   */
+  void recordException(T span, String message, RuntimeException runtimeException, ErrorCode errorCode);


Can we use Exception instead of RuntimeException to make this more generic?

Also, this method seems similar to addEvent. It seems more like a convenience method. We could define the interface in a different way that would make it more generic and allow flexibility in the event implementation. If we remove recordException but maintain addEvent we could have

public void addEvent(T span, Event event); public interface Event {} public class ExceptionEvent implements Event { private final String message; private final Exception exception; .... } public class GenericEvent implements event { private final String eventName; .... }

Hi @ZacBlanco Thanks for the feedback. Updated accordingly.

ZacBlanco · 2025-03-07T23:35:57Z

RFC-0011-open-telemetry.md

+   *
+   * @return the invalid span
+   */
+  T getInvalidSpan();


Why do we need a different method for this? If tracing is disabled, the implementation should handle returning an implementation of T which is an invalid span. Otherwise, users would need to write a conditional every time this is called?

I think yes the method is needed @ZacBlanco
If plugin loaded & tracing disabled --> no-op sdk span
If plugin not loaded & tracing enabled/disabled --> default implementation of .

ZacBlanco · 2025-03-07T23:36:30Z

RFC-0011-open-telemetry.md

+   *
+   * @return the root span
+   */
+  T getRootSpan(String traceId);


Does it always create it? Or only if it doesn't exist yet?

For each query 1 root span will always be created.

ZacBlanco · 2025-03-07T23:36:48Z

RFC-0011-open-telemetry.md

+   * Creates and returns the ScopedSpan with input name.
+   *
+   * @param name     the name
+   * @param skipSpan the skip span


what is a `skipSpan?

skipSpan corresponds to span-sampling in telemetry-tracing.properties to skip the spans which are less required. These were added to reduce the number of spans so that we can improve the performance and increase throughput, especially in high-load environments.

ZacBlanco · 2025-03-07T23:37:04Z

RFC-0011-open-telemetry.md

+   * @param span the span
+   * @return the optional
+   */
+  Optional<String> spanString(T span);


Why is there a separate method for this? Why can't this just be the implementation of T.toString()?

Thanks for the feedback. Moved the implementation.

sureshbabu-areekara force-pushed the main branch from d133671 to 8c51e68 Compare November 8, 2024 04:53

sureshbabu-areekara mentioned this pull request Nov 8, 2024

Enhance Open Telemetry Implementation in Presto prestodb/presto#23975

Open

tdcmeehan requested changes Nov 8, 2024

View reviewed changes

sureshbabu-areekara force-pushed the main branch from 2a53277 to 286b13c Compare November 13, 2024 16:48

tdcmeehan added the from:IBM PRs from IBM label Nov 13, 2024

ethanyzhang added from:IBM PRs from IBM and removed from:IBM PRs from IBM labels Nov 19, 2024

sureshbabu-areekara mentioned this pull request Nov 25, 2024

OTel Tracing implementation prestodb/presto#24133

Open

6 tasks

aaneja reviewed Mar 3, 2025

View reviewed changes

aaneja suggested changes Mar 3, 2025

View reviewed changes

sureshbabu-areekara added 4 commits March 4, 2025 12:51

RFC for Open Telemetry Implementation for Presto.

e1aafef

Updated Adoption plan.

b55d98d

Added low level design.

fbfbf49

Updated as per the latest implementation.

de67f6e

sureshbabu-areekara force-pushed the main branch from 5ab4ff2 to de67f6e Compare March 4, 2025 08:54

Review comments.

a1c882c

sureshbabu-areekara force-pushed the main branch from 621e4dc to a1c882c Compare March 4, 2025 18:16

ZacBlanco requested changes Mar 4, 2025

View reviewed changes

sureshbabu-areekara force-pushed the main branch from 62ca790 to cb99153 Compare March 6, 2025 18:02

Review comments.

2cf8a69

sureshbabu-areekara force-pushed the main branch from cb99153 to 2cf8a69 Compare March 7, 2025 07:04

ZacBlanco reviewed Mar 7, 2025

View reviewed changes

Review comments.

63be222

sureshbabu-areekara force-pushed the main branch from a2b105f to 63be222 Compare March 11, 2025 04:47


		*exporter-timeout*: how long the span exporter will wait for a batch of spans to be successfully sent before timing out
		*trace-sampling-ratio*: Double between 0.0 and 1.0 to specify the percentage of queries to be traced.


		OpenTelemetry is a powerful serviceability framework that helps to gain insights into the performance and behaviour of the systems. It facilitates generation, collection, and management of telemetry data such as traces.

		The OSS Presto had a basic implementation of Open Telemetry.


		The OSS Presto had a basic implementation of Open Telemetry.

		![Traces existing implementation](/RFC-0009-open-telemetry/traces-existing-implementation-oss-presto.png)


		## Proposed Implementation

		Presto can be manually instrumented and will have the following advantages.

	Presto can be manually instrumented and will have the following advantages.
	With additional changes to the OpenTelemetry SPI, we get the following advantages -

		@@ -67,9 +67,16 @@ OpenTelemetry maintains several official propagators. The default propagator is
		Based on the discussion, this may need to be updated with feedback from reviewers.

		## Adoption Plan



		## Summary
		The existing Open Telemetry implementation https://github.com/prestodb/presto/pull/18534 was an experimental feature, had a limited set of telemetry data(Query state changes) and did not include a child span concept. The recent implementation will make Presto more flexible, allowing support for both parent and child spans. Additionally, traces can now be propagated to the worker nodes as well.

-The existing Open Telemetry implementation https://github.com/prestodb/presto/pull/18534 was an experimental feature, had a limited set of telemetry data(Query state changes) and did not include a child span concept. The recent implementation will make Presto more flexible, allowing support for both parent and child spans. Additionally, traces can now be propagated to the worker nodes as well.
+Presto's original tracing implementation was merged in https://github.com/prestodb/presto/pull/18534. It was an experimental feature, had a limited set of telemetry data, and did not include a child span concept. This document proposes an expansion to the tracing provider SPI to
+. Support child spans in traces to make understanding nested operations simpler
+. Allow traces to be propagated to workers so that the entire lifespan of a query can be traced from analysis through execution



		## Background
		OpenTelemetry is a powerful serviceability framework that helps to gain insights into the performance and behaviour of the systems. It facilitates generation, collection, and management of telemetry data such as traces.

		OpenTelemetry is a powerful serviceability framework that helps to gain insights into the performance and behaviour of the systems. It facilitates generation, collection, and management of telemetry data such as traces.


		### Existing Open Telemetry

	The OSS Presto had a basic implementation of Open Telemetry with few limitations listed below.
	Presto has a basic `TracerProvider` interface which makes it difficult to implement the following desirable features:



		### Configuration
		Trace can be configured by modifying the values in presto-main/etc/telemetry-tracing.properties

	- Does not identify failed queries. No suitable methods in SPI to pass Error/Exception information and deals with failed query spans.
	- Does not identify failed queries. No suitable methods in SPI to pass Error/Exception information and details with failed query spans.

RFC for Open Telemetry Implementation for Presto. #33

Are you sure you want to change the base?

RFC for Open Telemetry Implementation for Presto. #33

Uh oh!

Conversation

sureshbabu-areekara commented Nov 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sureshbabu-areekara Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prestodb-ci commented Nov 19, 2024

Uh oh!

sureshbabu-areekara commented Nov 25, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sureshbabu-areekara Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZacBlanco Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

sureshbabu-areekara Nov 18, 2024 •

edited

Loading

sureshbabu-areekara Mar 4, 2025 •

edited

Loading

ZacBlanco Mar 4, 2025 •

edited

Loading

ZacBlanco Mar 4, 2025 •

edited

Loading