query execution fails if query pod deleted during query is running

[jason-heo]

Hello,

I'm currently using Apache Spark mainly since 2016 and now considering adopting Databend.

I have deployed a Databend cluster on a managed Kubernetes (k8s) cluster within my team. The performance of Databend has been pretty good so far.

However, I am encountering an issue. When I intentionally delete a query pod while a query is running, I receive the following error:

error: APIError:
ResponseError with 1067:
h2 protocol error:
error reading a body from connection: stream closed because of a broken pipe, source:

Some(
  hyper::Error(
    Body,
    Error {
      kind: Io(
        Custom {
          kind: BrokenPipe,
          error: "stream closed because of a broken pipe"
        }
     )
    }
  )
)

I am wondering if this is expected behavior (i.e., I need to re-run the failed query) or if there is a tunable parameter that could handle this scenario.

In Apache Spark, for instance, a SQL is divided into multiple tasks, and a task failure triggers automatic retries up to spark.task.maxFailures - 1 times.

Thank you for your assistance.

[sundy-li]

We can support query_max_failures settings to retry the query when encountering the retriable errors (network etc)

[jason-heo]

@sundy-li

Hello.

That's good news to hear. I'm looking forward to that setting.

I have a question.

If I understand you correctly, a failed query will restarted after retriable errors which means successfully evaluated records will be re-evaluated again due to entire query task will be retried.

Is my understanding correct?

Spark's job is consist of so called task, and trino has split. When failed, only failed tasks or splits are retried.

Is this possible in Databend?

Thanks.

[sundy-li]

It's not easy to implement partial retries in Databend. We can support full SQL retry firstly and then consider optimizing it to be partial later.

[jason-heo]

Thank you for your explanation :)

will watch #16771

closing this.