Python: Bug: Missing parameters when files are created inside Code Interpreter using Azure Responses API

[ymuichiro]

I confirmed that Code Interpreter can run via Azure Responses API in Semantic Kernel, but I found two issues: an error occurs when a file is created inside the Code Interpreter, and the response lacks parameters required to access the created file.

Bug 1: ValidationError for AnnotationContent

Semantic Kernel uses the following enum for AnnotationContent.content_type:

@experimental
class CitationType(str, Enum):
    """Citation type."""

    URL_CITATION = "url_citation"
    FILE_PATH = "file_path"
    FILE_CITATION = "file_citation"

However, the Responses API returns file citations with a type like this:

{"container_id": "cntr_68a6c50135d88190b6fdc062051155b50847ddec73d3be1b", "end_index": 94, "file_id": "cfile_68a6c67a7fcc8190889d7e1789d485e2", "filename": "sample.txt", "start_index": 66, "type": "container_file_citation"}

There is no matching enum member on the Semantic Kernel side. The CitationType should include CONTAINER_FILE_CITATION, for example:

@experimental
class CitationType(str, Enum):
    """Citation type."""

    URL_CITATION = "url_citation"
    FILE_PATH = "file_path"
    FILE_CITATION = "file_citation"
    CONTAINER_FILE_CITATION = "container_file_citation"

Bug 2: Missing parameters to access generated files

The response returned by the agent shows a file_id, but lacks the container_id and filename required to download the file (the filename is needed to replace sandbox:/mnt/data/sample.txt):

Example call:

r = await agent.get_response(cast(list, history.messages), thread=thread)
print(r.content.items)

Example output:

[
  TextContent(inner_content=None, ai_model_id=None, metadata={}, content_type='text', text='<user_prompt>', encoding=None),
  TextContent(inner_content=None, ai_model_id=None, metadata={}, content_type='text', text='<assistant_message>\n\n[sample.txt](sandbox:/mnt/data/sample.txt)', encoding=None), 
  AnnotationContent(inner_content=None, ai_model_id=None, metadata={}, content_type='annotation', file_id='cfile_68a6c67a7fcc8190889d7e1789d485e2', quote=None, start_index=88, end_index=116, url=None, title=None, citation_type=<CitationType.CONTAINER_FILE_CITATION: 'container_file_citation'>)
]

To include these values in the parsed result, AnnotationContent in semantic_kernel/contents/annotation_content.py should expose container_id and filename, and map the incoming type to citation_type. Example suggested structure:

@experimental
class AnnotationContent(KernelContent):
    """Annotation content."""

    content_type: Literal[ContentTypes.ANNOTATION_CONTENT] = Field(ANNOTATION_CONTENT_TAG, init=False)  # type: ignore
    tag: ClassVar[str] = ANNOTATION_CONTENT_TAG
    file_id: str | None = None
    quote: str | None = None
    start_index: int | None = None
    end_index: int | None = None
    url: str | None = None
    title: str | None = None
    # added
    container_id: str | None = None
    filename: str | None = None
    citation_type: CitationType | None = Field(None, alias="type")

    model_config = ConfigDict(
        extra="ignore",
        populate_by_name=True,
    )

    def __str__(self) -> str:
        ...

    def to_element(self) -> Element:
        ...

    @classmethod
    def from_element(cls: type[_T], element: Element) -> _T:
        ...

    def to_dict(self) -> dict[str, Any]:
        ...

Summary: add the missing enum value container_file_citation and extend AnnotationContent to include container_id and filename (and map type to citation_type) so file references returned by the Azure Responses API are fully usable.

Code

client: AsyncAzureOpenAI = AsyncAzureOpenAI(
    base_url=urljoin(server_settings.AZURE_OPENAI_COMPLETION_ENDPOINT.rstrip("/") + "/", "openai/v1/"),
    api_key=server_settings.AZURE_OPENAI_COMPLETION_API_KEY,
    api_version="preview",
)

async def run(history: ChatHistory) -> dict[str, Any]:
    with open("sample.txt", "rb") as fp:
        f = await client.files.create(file=fp, purpose="assistants")

    agent = AzureResponsesAgent(
        ai_model_id=server_settings.AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME,
        client=client,
        name="name",
        function_choice_behavior=FunctionChoiceBehavior.Required(),
        tools=[
            CodeInterpreter(
                type="code_interpreter",
                container=CodeInterpreterContainerCodeInterpreterToolAuto(type="auto", file_ids=[f.id]),
            )
        ],
    )

    thread = ResponsesAgentThread(client, history, previous_response_id=None, enable_store=True)
    r = await agent.get_response(cast(list, history.messages), thread=thread)
    print(thread.id, r.content.items)

    annotation_content = [item for item in r.content.items if isinstance(item, AnnotationContent)]

    for ac in annotation_content:
        print(ac.file_id, ac.filename, ac.container_id)

Platform

• Language: Python
• Source: pip main branch
• AI model: Azure OpenAI Responses API
• IDE: VS Code
• OS: Mac

[ymuichiro]

PR: #12985

[moonbox3]

Thanks!