[receiver/databricksreceiver] add skeleton and initial README #20891
Conversation
| [cluster.id, spark.app.id, pipeline.id, pipeline.name] | ||
| databricks.spark.block_manager.memory.off_heap.max: | ||
| enabled: true | ||
| description: n/a |
There was a problem hiding this comment.
Will we keep it n/a going forward?
There was a problem hiding this comment.
Unfortunately, not even Spark documents these metrics. It would be nice to have a description here but I'm not sure where to get this information.
| unit: "{jobs}" | ||
| gauge: | ||
| value_type: int | ||
| databricks.jobs.schedule.status: |
There was a problem hiding this comment.
Is this metric just an aggregation of databricks.tasks.schedule.status? If that's the case, I have a couple of Qs:
- Do we need to expose them both? If so, we should make one of them optional, otherwise, we expose duplicated data (just in different forms). Any backend (or collector) can do the aggregation.
- If the aggregation method is sum, they should be sums, not gauges.
There was a problem hiding this comment.
The status comes from the job, and a job has multiple tasks, so the job status is simply propagated to each of its tasks. I suppose the task status metric could be made optional or removed altogether. What do you think?
|
Sure, we need a rebase and we can get this in. The receiver needs more codeowners and probably someone from Databricks to help out if anyone is interested. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
@MovieStoreGuy rebased, ready to go in |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
|
Any updates here? @atoulme |
|
Not much - we still want to do it, just no time. Interested? |
Description: Adds DataBricks receiver skeleton.
Link to tracking Issue: #18724
Testing: Unit tests added.
Documentation: Initial README added.