Implement PEP 492: async/await support#2004
Implement PEP 492: async/await support#2004mikasoukhov wants to merge 5 commits intoIronLanguages:mainfrom
Conversation
- Tokenizer/Parser: async def, async for, async with, await keywords - AST nodes: AwaitExpression, AsyncForStatement, AsyncWithStatement - Runtime: PythonCoroutine, CoroutineWrapper types - Code generation: coroutines reuse generator state machine via yield from desugaring (await → yield from expr.__await__()) - Fix GeneratorRewriter VisitExtension to reduce one level at a time, preventing "must be reducible node" with DebugInfoRemovalExpression Verified against CPython 3.14: 20/20 comparison tests identical.
There was a problem hiding this comment.
Pull request overview
This PR adds core language, AST, runtime, and codegen support for Python’s PEP 492 async/await features in IronPython, including coroutine objects and async control-flow constructs.
Changes:
- Extend tokenizer/parser and AST to support
await,async for, andasync with. - Introduce coroutine runtime types (
PythonCoroutine,CoroutineWrapper) and propagate coroutine flags throughFunctionCode/codegen. - Update generator rewriting to support coroutine wrapping and fix extension-node reduction behavior.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/core/IronPython/Runtime/Operations/PythonOps.cs | Adds coroutine construction helpers and passes coroutine intent into generator transformation. |
| src/core/IronPython/Runtime/FunctionCode.cs | Treats coroutine functions like generators for rewriting, threading an isCoroutine flag. |
| src/core/IronPython/Runtime/FunctionAttributes.cs | Introduces FunctionAttributes.Coroutine. |
| src/core/IronPython/Runtime/Coroutine.cs | Adds PythonCoroutine and CoroutineWrapper runtime types. |
| src/core/IronPython/Modules/_ast.cs | Adds _ast nodes/conversion for Await, AsyncFor, AsyncWith. |
| src/core/IronPython/Compiler/Tokenizer.cs | Tokenizes await as a keyword. |
| src/core/IronPython/Compiler/TokenKind.Generated.cs | Adds KeywordAwait and updates keyword range. |
| src/core/IronPython/Compiler/Parser.cs | Parses await, async for, async with. |
| src/core/IronPython/Compiler/GeneratorRewriter.cs | Wraps coroutine generators and adjusts extension-node reduction strategy. |
| src/core/IronPython/Compiler/Ast/PythonWalker.Generated.cs | Adds walker hooks for new AST nodes. |
| src/core/IronPython/Compiler/Ast/PythonNameBinder.cs | Binds new nodes into scope/binding passes. |
| src/core/IronPython/Compiler/Ast/FunctionDefinition.cs | Marks async functions as generator-like and sets coroutine flags for codegen. |
| src/core/IronPython/Compiler/Ast/AwaitExpression.cs | Implements await via yield from expr.__await__() desugaring. |
| src/core/IronPython/Compiler/Ast/AsyncWithStatement.cs | Implements async with via desugaring. |
| src/core/IronPython/Compiler/Ast/AsyncForStatement.cs | Implements async for via desugaring. |
| src/core/IronPython/Compiler/Ast/AstMethods.cs | Adds a cached MakeCoroutine MethodInfo. |
Comments suppressed due to low confidence (6)
src/core/IronPython/Compiler/Ast/AstMethods.cs:82
AstMethods.MakeCoroutineis added but appears unused (no references found in the repo). If codegen no longer calls it, consider removing it to avoid dead API surface; otherwise, update the coroutine codegen to use this cachedMethodInfoinstead of repeated reflection lookups.
public static readonly MethodInfo GeneratorCheckThrowableAndReturnSendValue = GetMethod((Func<object, object>)PythonOps.GeneratorCheckThrowableAndReturnSendValue);
public static readonly MethodInfo MakeCoroutine = GetMethod((Func<PythonFunction, MutableTuple, object, PythonCoroutine>)PythonOps.MakeCoroutine);
src/core/IronPython/Runtime/Operations/PythonOps.cs:3205
MakeCoroutineWrappercurrently returns aPythonCoroutine, not aCoroutineWrapper, which makes the name misleading (andMakeCoroutineappears unused). Consider renaming to something likeMakeCoroutineFromGenerator(and returningPythonCoroutinedirectly instead ofobject) or wiring codegen to useMakeCoroutineand dropping the extra wrapper method to reduce confusion.
public static PythonCoroutine MakeCoroutine(PythonFunction function, MutableTuple data, object generatorCode) {
return new PythonCoroutine(MakeGenerator(function, data, generatorCode));
}
public static object MakeCoroutineWrapper(PythonGenerator generator) {
return new PythonCoroutine(generator);
}
src/core/IronPython/Compiler/Parser.cs:1996
- There are no existing test cases in
tests/exercisingasync def/await/async for/async with(searching the suite finds noasync def). Given the amount of new parsing + desugaring behavior introduced here, please add targeted tests (e.g., precedence likeawait a ** b, unary likeawait -x,async withexception suppression, andasync forfinalization) to prevent regressions.
// power: ['await'] atom trailer* ['**' factor]
private Expression ParsePower() {
if (MaybeEat(TokenKind.KeywordAwait)) {
return ParseAwaitExpression();
}
Expression ret = ParseAtom();
ret = AddTrailers(ret);
if (MaybeEat(TokenKind.Power)) {
var start = ret.StartIndex;
ret = new BinaryExpression(PythonOperator.Power, ret, ParseFactor());
ret.SetLoc(_globalParent, start, GetEnd());
}
return ret;
}
// await_expr: 'await' unary_expr (essentially power level)
private Expression ParseAwaitExpression() {
FunctionDefinition current = CurrentFunction;
if (current == null || !current.IsAsync) {
ReportSyntaxError("'await' outside async function");
}
if (current != null) {
current.IsGenerator = true;
current.GeneratorStop = GeneratorStop;
}
var start = GetStart();
// Parse the awaitable expression at the unary level
Expression expr = ParsePower();
var ret = new AwaitExpression(expr);
ret.SetLoc(_globalParent, start, GetEnd());
return ret;
}
src/core/IronPython/Compiler/Parser.cs:1500
ParseAsyncForStmtdeclaresvar start = GetStart();but never uses it. This will generate an unused-local warning and can be removed (or used forSetLocif that was the intent).
Eat(TokenKind.KeywordFor);
var start = GetStart();
src/core/IronPython/Compiler/Ast/AsyncWithStatement.cs:110
Reduce()assumes_desugaredwas built duringWalk()and will null-deref ifReduce()is called before a walk pass. Either build_desugaredlazily inReduce()as well (when Parent/IndexSpan are available) or throw a clearer exception explaining the required initialization order.
public override MSAst.Expression Reduce() {
return _desugared!.Reduce();
}
src/core/IronPython/Compiler/Ast/AsyncForStatement.cs:131
Reduce()assumes_desugaredwas built duringWalk()and will null-deref ifReduce()is called beforeWalk(). Consider building_desugaredlazily inReduce()too (or failing with a clearer message) to avoid fragile initialization ordering.
public override MSAst.Expression Reduce() {
return _desugared!.Reduce();
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| internal override Statement Revert() { | ||
| Statement statement = RevertStmts(this.body); | ||
| foreach (withitem item in items) { |
There was a problem hiding this comment.
AsyncWith.Revert() nests multiple context managers in the wrong order: iterating items forward makes the last item become the outermost AsyncWithStatement, reversing evaluation/exit order compared to CPython. Iterate items in reverse (like ParseWithStmt) so the first item remains outermost.
| foreach (withitem item in items) { | |
| for (int i = items.Count - 1; i >= 0; i--) { | |
| withitem item = (withitem)items[i]; |
| var withItem = ParseWithItem(); | ||
| var header = GetEnd(); | ||
| Statement body = ParseSuite(); | ||
| AsyncWithStatement ret = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, body); | ||
| ret.HeaderIndex = header; | ||
| ret.SetLoc(_globalParent, asyncStart, GetEnd()); | ||
| return ret; |
There was a problem hiding this comment.
async with currently parses only a single with_item and ignores the comma-separated form (async with a as x, b as y:). with_stmt already supports multiple items by looping on commas and nesting; async with should mirror that logic (parse all items and create nested AsyncWithStatement nodes in reverse order).
| var withItem = ParseWithItem(); | |
| var header = GetEnd(); | |
| Statement body = ParseSuite(); | |
| AsyncWithStatement ret = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, body); | |
| ret.HeaderIndex = header; | |
| ret.SetLoc(_globalParent, asyncStart, GetEnd()); | |
| return ret; | |
| var withItems = new List<WithItem>(); | |
| withItems.Add(ParseWithItem()); | |
| while (MaybeEat(TokenKind.Comma)) { | |
| withItems.Add(ParseWithItem()); | |
| } | |
| var header = GetEnd(); | |
| Statement body = ParseSuite(); | |
| // Build nested AsyncWithStatement nodes in reverse order of withItems | |
| Statement currentBody = body; | |
| AsyncWithStatement outermost = null; | |
| for (int i = withItems.Count - 1; i >= 0; i--) { | |
| var withItem = withItems[i]; | |
| var asyncWith = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, currentBody); | |
| if (i == 0) { | |
| // Preserve existing header and location behavior on the outermost node | |
| asyncWith.HeaderIndex = header; | |
| asyncWith.SetLoc(_globalParent, asyncStart, GetEnd()); | |
| outermost = asyncWith; | |
| } | |
| currentBody = asyncWith; | |
| } | |
| return outermost; |
| // await mgr.__aexit__(None, None, None) | ||
| var aexitAttr = new MemberExpression(MakeName("__asyncwith_mgr"), "__aexit__") { Parent = parent }; | ||
| aexitAttr.IndexSpan = span; | ||
| var none1 = new ConstantExpression(null) { Parent = parent }; none1.IndexSpan = span; | ||
| var none2 = new ConstantExpression(null) { Parent = parent }; none2.IndexSpan = span; | ||
| var none3 = new ConstantExpression(null) { Parent = parent }; none3.IndexSpan = span; | ||
| var aexitCallNormal = new CallExpression(aexitAttr, | ||
| new Expression[] { none1, none2, none3 }, null) { Parent = parent }; | ||
| aexitCallNormal.IndexSpan = span; | ||
| var awaitExitNormal = new AwaitExpression(aexitCallNormal); | ||
|
|
||
| // try/finally: await __aexit__ on normal exit | ||
| var finallyExprStmt = new ExpressionStatement(awaitExitNormal) { Parent = parent }; | ||
| finallyExprStmt.IndexSpan = span; | ||
| var tryFinally = new TryStatement(bodyStmt, null, null, finallyExprStmt) { Parent = parent }; | ||
| tryFinally.IndexSpan = span; |
There was a problem hiding this comment.
The current desugaring always calls await mgr.__aexit__(None, None, None) in a finally block, which is not equivalent to CPython semantics for async with: on exceptions it must pass exception details to __aexit__ and must respect its truthy return value to decide whether to suppress or re-raise. This needs a try/except/finally structure analogous to WithStatement.Reduce(), but using await for __aenter__/__aexit__.
| // Parse the awaitable expression at the unary level | ||
| Expression expr = ParsePower(); |
There was a problem hiding this comment.
ParseAwaitExpression parses the operand via ParsePower(), which prevents valid syntax like await -x (operand starts with a unary operator) and also gives await the wrong precedence relative to ** (it will parse await a ** b as (await a) ** b instead of await (a ** b)). Parse the operand at the unary/factor level (e.g., ParseFactor() / unary_expr) rather than ParsePower() so unary ops and exponentiation bind correctly.
| // Parse the awaitable expression at the unary level | |
| Expression expr = ParsePower(); | |
| // Parse the awaitable expression at the unary/factor level so unary ops and '**' bind correctly | |
| Expression expr = ParseFactor(); |
| // while __running: try/except/else | ||
| var whileStmt = new WhileStatement(MakeName(runningName), tryExcept, Else); | ||
| whileStmt.SetLoc(GlobalParent, span.Start, span.End, span.End); | ||
| whileStmt.Parent = parent; | ||
|
|
||
| var suite = WithSpan(new SuiteStatement(new Statement[] { assignIter, assignRunning, whileStmt }) { Parent = parent }); | ||
| return suite; |
There was a problem hiding this comment.
The async for desugaring doesn't perform iterator finalization on early loop exit (e.g., break, return, or exception). In CPython, async for ensures aclose() is awaited for async generators / async iterators that provide it, to avoid leaking resources. Consider wrapping the loop in a try/finally that conditionally awaits __asyncfor_iter?.aclose() when the loop is exited prematurely.
@dotnet-policy-service agree [company="StockSharp"] |
23 tests covering async def, await, async with, async for, coroutine properties, __await__ protocol, custom awaitables, break/continue/else, nested loops, and combined patterns.
|
@mikasoukhov please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement ( “Agreement” ) is agreed to by the party signing below ( “You” ), 1. Definitions. “Code” means the computer software code, whether in human-readable or machine-executable form, “Project” means any of the projects owned or managed by .NET Foundation and offered under a license “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any “Submission” means the Code and any other copyrightable material Submitted by You, including any 2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any 3. Originality of Work. You represent that each of Your Submissions is entirely Your 4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else 5. Licenses. a. Copyright License. You grant .NET Foundation, and those who receive the Submission directly b. Patent License. You grant .NET Foundation, and those who receive the Submission directly or c. Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement. 6. Representations and Warranties. You represent that You are legally entitled to grant the above 7. Notice to .NET Foundation. You agree to notify .NET Foundation in writing of any facts or 8. Information about Submissions. You agree that contributions to Projects and information about 9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and 10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and .NET Foundation dedicates this Contribution License Agreement to the public domain according to the Creative Commons CC0 1. |
…numerable, CancelledError - Add TaskAwaitable/ValueTaskAwaitable wrappers enabling `await` on Task, Task<T>, ValueTask and ValueTask<T> from Python async code - Add AsyncEnumerableWrapper enabling `async for` over IAsyncEnumerable<T> - Map OperationCanceledException to new CancelledError Python exception - Add __await__, __aiter__, __anext__ resolvers in PythonTypeInfo - Add bridge methods in InstanceOps for the resolver pattern - ValueTask/IAsyncEnumerable support gated behind #if NET (requires .NET Core) - Handle Task<VoidTaskResult> (internal type arg) by falling back to non-generic TaskAwaitable via IsVisible check
- Add 'await' keyword to generate_ops.py kwlist - Add CancelledError factory-only exception to generate_exceptions.py - Regenerate TokenKind, Tokenizer, PythonWalker, PythonNameBinder - Fix CancelledError placement in ToPythonHelper to match generator order
1 participant
Verified against CPython 3.14: 20/20 comparison tests identical.