[BLOG] Monitoring Apache Spark using opensearch

[avichaym]

Describe the blog post your would like to write

Opensearch can be an ideal choice for continuous monitoring of apache spark jobs in production.
This year , we released the Guidance for Analytics Observability on AWS reference solution , which enhances Any spark ditro (OSS, Glue , EMR) with a plugin JAR that outputs various spark metrics to Opensearch via Data Prepper. This solution also provides multiple operational dashboards for apache spark for root causing common issues like Data Skew , and also provides a Vega visualization for historical Spark-SQL plans as a Graph.

The blog will also demonstrate how opensearch anomaly detection features can further enhance the spark monitoring capabilities.

What is the title of the blog post?
Monitoring Apache Spark using opensearch

Who are the authors?
Avichay Marciano, Vincent Gromakowsky

What is the proposed posting date?
September 1st

[pajuric]

Thank for the blog idea. When you are ready, please feel free to load a draft of the blog here or send me a copy to pattijur@amazon.com.

[pajuric]

@avichaym - Following up to see if you still intend to write and contribute this blog, or if we are able to close this issue? Please advise.

[pajuric]

@avichaym - Moving this into backlog and awaiting input from the the author about whether they plan to m0ve this forward.