Blog post on query cancellation #75

pepijnve · 2025-06-15T15:15:49Z

Blog post describing the problem solved by and the changes made in 16398

Closes Blog post about DataFusion Async / Stream execution model / cancellation datafusion#16396

pepijnve · 2025-06-15T15:19:10Z

I've marked this as draft for now. I think I have the narrative arc I was going for in place, but the text probably still needs some editing work.

djanderson

I found this write-up very helpful and clear. Just a couple small typo suggestions and 1 clarifying question:

content/blog/2025-06-15-cancellation.md

djanderson · 2025-06-16T21:20:03Z

content/blog/2025-06-15-cancellation.md

+When calling `Receiver::recv` for one of these channels, a unit of Tokio task budget is consumed.
+As a consequence, query plans that made use of exchange-like operators were already mostly cancelable.
+
+### Depleting The Budget


I was looking for docs on poll_proceed and made_progress and I'm wondering if those functions have been renamed to has_budget_remaining and consume_budget recently? https://docs.rs/tokio/latest/tokio/task/coop/index.html#functions

Or, am I looking in the wrong place?

poll_proceed/made_progress are the functions Tokio itself uses to consume budget in resources like Channel. consume_budget is implemented using those two as well.

At the moment they're still pub(crate) in tokio. I'm working on a PR to make them accessible at tokio-rs/tokio#7405. I wrote this post under the assumption that this will get merged. We shouldn't publish this post until that actually lands.

The DataFusion PR apache/datafusion#16398 currently contains three variants: one with the manual counter, one using has_budget_remaining/consume_budget as approximation, and one using poll_proceed/made_progress which doesn't compile just yet.

Thanks for the context!

djanderson · 2025-06-16T21:50:20Z

One other thing I'm curious about. This write-up discusses the change in terms of enabling long-running tasks to be cancelled, but would making CPU-intensive exec blocks more cooperative also help alleviate blocking IO on the main runtime if users don't set up a separate runtime ala apache/datafusion#16331? That could be a really nice benefit besides cancelability of this, if so. @alamb?

pepijnve · 2025-06-17T11:52:55Z

but would making CPU-intensive exec blocks more cooperative also help alleviate blocking IO on the main runtime if users don't set up a separate runtime

It should make some more room for the IO indeed. Cancellation is just one facet of the more general problem of fairness in a cooperatively scheduled runtime. Yielding every now and then helps to ensure every concurrent task gets a fair share of time to run.

pepijnve · 2025-06-17T13:15:12Z

A colleague of mine suggested to add some more diagrams. I know UML is out of fashion, but here's a sketch of what's actually happening when you imagine Tokio's budget is initialized to 1. Would something like this be a useful addition, or trying to cram too much information in a small space?

alamb · 2025-06-17T16:45:26Z

A colleague of mine suggested to add some more diagrams. I know UML is out of fashion, but here's a sketch of what's actually happening when you imagine Tokio's budget is initialized to 1. Would something like this be a useful addition, or trying to cram too much information in a small space?

I think it is hugely helpful (we just shouldn't call it UML to avoid getting painted with the "old folks brush" 😆 )

alamb · 2025-06-17T16:46:15Z

FYI @ozankabak and @zhuqi-lucas

alamb · 2025-06-17T16:50:12Z

content/blog/2025-06-15-cancellation.md

@@ -0,0 +1,353 @@
+---


I took the liberty of pushing this "front matter" and ASF header to this post

I tried to render the blog locally to preview but found it wasn't in the blog list without this

Since Datadobi is paying for my time while I work on this stuff, would it be possible/allowed to add a small note mentioning that?

Yes of course -- we have done this on other blogs as well

Something like "Thank you to Datadobi for supporting this work" would be very much following the same pattern

alamb

This looks great to me -- thank you @pepijnve

I am approving as I think the post could be posted as is, but of course we can also make it better.

I left a bunch of suggestions, but I also happy to help implement them if you prefer. Sorry I got carried away but I really like this post and thank you so much for writing it

alamb · 2025-06-17T16:56:09Z

content/blog/2025-06-15-cancellation.md

+## The Challenge of Cancelling Long-Running Queries
+
+Have you ever tried to cancel a query that just wouldn't stop?
+In this post, we'll take a look at why that can happen in DataFusion and what the community did to resolve the problem in depth.


I think it would also help here in the intro to emphasize other learnings a potential reader might get (to convince them to read more):

Review the async model / tokio tasks

Learn how DataFusion uses that model for CPU bound work

Learn how query engines in general, and DataFusion specifically implement cooperative scheduling and cancelling

So manybe something like

Suggested change

In this post, we'll take a look at why that can happen in DataFusion and what the community did to resolve the problem in depth.

In this post, we'll review how Rust's `async` model works, and how DataFusion uses that model for CPU intensive tasks and how they implement cancellation. Then we'll review some cases where queries could not be cancelled in DataFusion and what the community did to resolve the problem.

alamb · 2025-06-17T16:58:48Z

content/blog/2025-06-15-cancellation.md

+
+### Understanding Rust's Async Model
+
+To really understand the cancellation problem you need to be somewhat familiar with Rust's asynchronous programming model.


Suggested change

To really understand the cancellation problem you need to be somewhat familiar with Rust's asynchronous programming model.

DataFusion, somewhat unconventionally, [uses the Rust async system and the tokio task pool](https://docs.rs/datafusion/latest/datafusion/#thread-scheduling-cpu--io-thread-pools-and-tokio-runtimes) for CPU intensive processing.

To really understand the the cancellation problem you need to understand the DataFusion execution model which and thus with Rust's asynchronous programming model.

alamb · 2025-06-17T17:02:19Z

content/blog/2025-06-15-cancellation.md

+
+#### Futures Are Inert
+
+Rust's asynchronous programming model is built around the `Future<T>` trait.


It might be great to add documentation links here

https://doc.rust-lang.org/std/future/trait.Future.html

https://www.javadoc.io/doc/io.trane/future-java/0.1.0/io/trane/future/Promise.html

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise

content/blog/2025-06-15-cancellation.md

alamb · 2025-06-20T11:37:54Z

One other thing I'm curious about. This write-up discusses the change in terms of enabling long-running tasks to be cancelled, but would making CPU-intensive exec blocks more cooperative also help alleviate blocking IO on the main runtime if users don't set up a separate runtime ala apache/datafusion#16331? That could be a really nice benefit besides cancelability of this, if so. @alamb?

@djanderson it may help, but I think even if the CPU intensive tasks yielded more frequently we would still need a separate runtime to avoid increasing io latnecy

alamb · 2025-06-20T11:43:24Z

@pepijnve I pushed an Acknowedgement and "About DataFusion" sections

alamb · 2025-06-23T14:57:28Z

Hi @pepijnve -- I wonder if you plan to work on this post in the near term? If not I will try and find time to add some diagrams / etc to help it get ready to publish (as I think it is important content)

Thanks again for the work so far

pepijnve · 2025-06-23T15:29:02Z

I was not planning on changing it substantially anymore. I was thinking of maybe rereading the text with a fresh pair of eyes and editing a sentence here or there, but that's it. Need to switch gears from open source contribution mode to building our own product for a bit.

alamb · 2025-06-23T20:02:19Z

I was not planning on changing it substantially anymore. I was thinking of maybe rereading the text with a fresh pair of eyes and editing a sentence here or there, but that's it. Need to switch gears from open source contribution mode to building our own product for a bit.

Makes sense -- thank you -- I'll do the same (with fresh eyes) and polish it up and get it ready to merge. Thank you

alamb · 2025-06-24T14:53:00Z

I pushed the images to this post, updated the publish date to June 30 (next Monday), and started doing some wordsmiting

I'll finish up my polishing today and then maybe try and highlight this draft in case others want a chance to review it. It is really neat. Thanks again @pepijnve

comphead

Thanks @pepijnve it is very nice structured

comphead · 2025-06-24T18:14:02Z

I was wondering if it makes sense to add challenges why async is challenging to cancel on low level but it probably would be noisy. But just in case this article shed the light on cancellation challenges including Rust ecosystem https://without.boats/blog/asynchronous-clean-up/

alamb · 2025-06-24T19:48:33Z

I was wondering if it makes sense to add challenges why async is challenging to cancel on low level but it probably would be noisy. But just in case this article shed the light on cancellation challenges including Rust ecosystem https://without.boats/blog/asynchronous-clean-up/

Man that post somewhat blew my mind (I only understood about 50% of it).

alamb · 2025-06-24T19:50:12Z

In my opinion this article does a pretty good job explaining the issues with cancellation, but it doesn't talk about async destructors which I agree are probably best left out of scope for this article

pepijnve · 2025-06-25T06:57:57Z

https://without.boats/blog/asynchronous-clean-up/

Wait till you read their work on Pin/Unpin.

pepijnve · 2025-06-25T07:03:59Z

I was wondering if it makes sense to add challenges why async is challenging to cancel on low level but it probably would be noisy. But just in case this article shed the light on cancellation challenges including Rust ecosystem https://without.boats/blog/asynchronous-clean-up/

Thanks for the pointer. I think this would be useful as hyperlink in the intro where we're trying to say "cancelling async ain't that simple". Another useful blog post to link to might be https://cybernetist.com/2024/04/19/rust-tokio-task-cancellation-patterns/

pepijnve · 2025-06-25T07:07:33Z

In my opinion this article does a pretty good job explaining the issues with cancellation, but it doesn't talk about async destructors which I agree are probably best left out of scope for this article

If we do link to it I would link directly to https://without.boats/blog/asynchronous-clean-up/#cancellation rather than the top of the article. The idea of a 'spectrum of cooperativeness' when it comes to cancellation is a very elegant way of describing the problem. DataFusion tasks were a mix of implicitly cooperative (the Receiver users) and non-cooperative. The work that was done tries to move all tasks towards the cooperative end of the spectrum.

pepijnve · 2025-06-25T14:57:00Z

@alamb I've done some work to make the diagrams a bit prettier using Excalidraw. Massaged the text a bit further. Tried to improve the flow from one section to the next.

ozankabak

LGTM, thank you @pepijnve

content/blog/2025-06-30-cancellation.md

alamb · 2025-06-26T13:17:45Z

😍

alamb · 2025-06-26T13:27:49Z

I went through the draft again and I think it is looking REALLY nice -- something to be proud of I think

I pushed one more commit with some wordsmithing obsession and new links ab0094c

I can't want to publish this. It is going to be so good

content/blog/2025-06-30-cancellation.md

alamb · 2025-06-30T11:23:41Z

The blog is live!
https://datafusion.apache.org/blog/2025/06/30/cancellation/

comphead · 2025-06-30T21:24:51Z

The blog is live! https://datafusion.apache.org/blog/2025/06/30/cancellation/

Published on Datafusion Linkedin resource

kevinjqliu · 2025-07-05T18:59:59Z

Just read the post. Coming back here to thank everyone for the contribution! I always learn so much from these.

I think it would be great to have a comment section for the blog posts. I opened #80 to discuss this idea

pepijnve mentioned this pull request Jun 15, 2025

Blog post about DataFusion Async / Stream execution model / cancellation apache/datafusion#16396

Closed

pepijnve marked this pull request as draft June 15, 2025 15:18

pepijnve force-pushed the cancellation branch from 5c6fe91 to 3aba849 Compare June 15, 2025 15:19

First draft

e6a6eb5

pepijnve force-pushed the cancellation branch from 3aba849 to e6a6eb5 Compare June 16, 2025 07:29

djanderson reviewed Jun 16, 2025

View reviewed changes

Corrections based on feedback so far

198d12c

alamb mentioned this pull request Jun 17, 2025

Use Tokio's task budget consistently, better APIs to support task cancellation apache/datafusion#16398

Merged

Add front matter and ASF header

667f570

alamb reviewed Jun 17, 2025

View reviewed changes

alamb approved these changes Jun 17, 2025

View reviewed changes

pepijnve added 2 commits June 17, 2025 21:33

Integrating feedback from @alamb

173c07c

Add short inline comments

91be0df

Add Acknowledgement and About DataFusion sections

0c5333d

alamb marked this pull request as ready for review June 23, 2025 20:02

alamb added 2 commits June 24, 2025 10:35

Add images

bc5ca6c

wordsmithing

6531630

wordsmith and format

fb69d39

alamb assigned alamb and unassigned alamb Jun 24, 2025

comphead approved these changes Jun 24, 2025

View reviewed changes

Prettier images. Text revisions.

5369558

ozankabak approved these changes Jun 25, 2025

View reviewed changes

Another round of proofreading by AI

4190c7e

alamb reviewed Jun 26, 2025

View reviewed changes

content/blog/2025-06-30-cancellation.md Outdated Show resolved Hide resolved

Obession: Wordsmithing for concision

ab0094c

Update content/blog/2025-06-30-cancellation.md

6917588

alamb reviewed Jun 30, 2025

View reviewed changes

content/blog/2025-06-30-cancellation.md Outdated Show resolved Hide resolved

Update content/blog/2025-06-30-cancellation.md

c9cf1cc

alamb merged commit 8713024 into apache:main Jun 30, 2025
1 check passed

	In this post, we'll take a look at why that can happen in DataFusion and what the community did to resolve the problem in depth.
	In this post, we'll review how Rust's `async` model works, and how DataFusion uses that model for CPU intensive tasks and how they implement cancellation. Then we'll review some cases where queries could not be cancelled in DataFusion and what the community did to resolve the problem.


		### Understanding Rust's Async Model

		To really understand the cancellation problem you need to be somewhat familiar with Rust's asynchronous programming model.

	To really understand the cancellation problem you need to be somewhat familiar with Rust's asynchronous programming model.
	DataFusion, somewhat unconventionally, [uses the Rust async system and the tokio task pool](https://docs.rs/datafusion/latest/datafusion/#thread-scheduling-cpu--io-thread-pools-and-tokio-runtimes) for CPU intensive processing.
	To really understand the the cancellation problem you need to understand the DataFusion execution model which and thus with Rust's asynchronous programming model.


		#### Futures Are Inert

		Rust's asynchronous programming model is built around the `Future<T>` trait.

Blog post on query cancellation #75

Blog post on query cancellation #75

Uh oh!

Conversation

pepijnve commented Jun 15, 2025 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pepijnve commented Jun 15, 2025

Uh oh!

djanderson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

djanderson commented Jun 16, 2025

Uh oh!

pepijnve commented Jun 17, 2025

Uh oh!

pepijnve commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Jun 17, 2025

Uh oh!

alamb commented Jun 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pepijnve Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Jun 20, 2025

Uh oh!

alamb commented Jun 20, 2025

Uh oh!

alamb commented Jun 23, 2025

Uh oh!

pepijnve commented Jun 23, 2025

Uh oh!

alamb commented Jun 23, 2025

Uh oh!

alamb commented Jun 24, 2025

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

comphead commented Jun 24, 2025

Uh oh!

alamb commented Jun 24, 2025

Uh oh!

alamb commented Jun 24, 2025

Uh oh!

pepijnve commented Jun 25, 2025

Uh oh!

pepijnve commented Jun 25, 2025

pepijnve commented Jun 15, 2025 •

edited by alamb

Loading

pepijnve commented Jun 17, 2025 •

edited

Loading

pepijnve Jun 17, 2025 •

edited

Loading

kevinjqliu commented Jul 5, 2025 •

edited

Loading