Cannot terminate request if StrOutputParser() is in chain

[jimmyparadm]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from flask import Flask, Response, stream_with_context
from langchain_ollama import ChatOllama
from langchain_core.runnables import RunnablePassthrough
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

app = Flask(__name__)

data = {
    "model": "deepseek-r1-1.5b:2-32",
    "base_url": "http://127.0.0.1:11434",
    "stop": ['<|never_trigger|>'],
    "temperature": 0.7 
}
chat_model = ChatOllama(**data)

messages = [
    ("system", "You are a helpful assistant."),
    ("human", "{input}")
]
promptTemplate = ChatPromptTemplate.from_messages(messages)

@app.route('/stream', methods=['POST'])
def stream_response():
    combined_chain = promptTemplate | chat_model | StrOutputParser()
    retrieval_chain = (
        RunnablePassthrough.assign(answer=combined_chain)
    ).with_config(run_name="retrieval_chain")

    def generate_context():
        question = "Write a paragraph about cat."
        try:
            for chunk in retrieval_chain.stream({"input": question}):
                yield "data:" + str(chunk) + "\n\n"
        except GeneratorExit:
            print('Exit')
            return
    
    return Response(stream_with_context(generate_context()), content_type="text/event-stream")

if __name__ == "__main__":
    app.run(port=5001, host='0.0.0.0')

Description

I am building a python flask web app using langchain and ollama. I want to make sure ollama can stop processing request when the client terminate the connection. With the code above, when client disconnects, it does not trigger the GeneratorExit block until ollama finish processing the request.

I tried removing StrOutputParser() from the chain and repeat the steps. When the client disconnects, the GeneratorExit block is triggered instantly and ollama also stops processing the request right away.

I test this using cmd with command below:

curl -X POST -H "Content-Type: application/json" http://localhost:5001/stream

After the stream starts, I use crtl+c to disconnect the connection.

System Info

System Information

OS: Windows
OS Version: 10.0.22631
Python Version: 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]

Package Information

langchain_core: 0.3.27
langchain: 0.3.7
langchain_community: 0.3.7
langsmith: 0.1.147
langchain_chroma: 0.2.0
langchain_docling: 0.2.0
langchain_elasticsearch: 0.3.0
langchain_experimental: 0.3.3
langchain_google_community: 2.0.4
langchain_huggingface: 0.1.2
langchain_ollama: 0.2.2
langchain_openai: 0.2.8
langchain_qdrant: 0.2.0
langchain_text_splitters: 0.3.2
langchain_together: 0.2.0

Optional packages not installed

langserve

Other Dependencies

aiohttp: 3.11.10
async-timeout: Installed. No version info available.
beautifulsoup4: 4.12.3
chromadb: 0.4.17
dataclasses-json: 0.6.4
db-dtypes: Installed. No version info available.
docling: 2.15.1
elasticsearch[vectorstore-mmr]: Installed. No version info available.
fastapi: 0.110.0
fastembed: 0.2.6
gapic-google-longrunning: Installed. No version info available.
google-api-core: 2.24.0
google-api-python-client: 2.159.0
google-auth-httplib2: 0.2.0
google-auth-oauthlib: 1.2.0
google-cloud-aiplatform: Installed. No version info available.
google-cloud-bigquery: Installed. No version info available.
google-cloud-bigquery-storage: Installed. No version info available.
google-cloud-contentwarehouse: Installed. No version info available.
google-cloud-core: 2.4.1
google-cloud-discoveryengine: Installed. No version info available.
google-cloud-documentai: Installed. No version info available.
google-cloud-documentai-toolbox: Installed. No version info available.
google-cloud-speech: Installed. No version info available.
google-cloud-storage: Installed. No version info available.
google-cloud-texttospeech: Installed. No version info available.
google-cloud-translate: Installed. No version info available.
google-cloud-vision: 3.9.0
googlemaps: Installed. No version info available.
grpcio: 1.69.0
httpx: 0.27.2
httpx-sse: 0.4.0
huggingface-hub: 0.23.3
jsonpatch: 1.33
langsmith-pyo3: Installed. No version info available.
numpy: 1.26.4
ollama: 0.4.4
openai: 1.59.9
orjson: 3.9.15
packaging: 23.2
pandas: 2.2.3
pyarrow: 15.0.2
pydantic: 2.10.5
pydantic-settings: 2.6.1
PyYAML: 6.0.2
qdrant-client: 1.12.1
requests: 2.32.3
requests-toolbelt: 1.0.0
sentence-transformers: 3.0.1
SQLAlchemy: 2.0.23
tenacity: 8.2.3
tiktoken: 0.7.0
tokenizers: 0.19.1
transformers: 4.44.2
typing-extensions: 4.12.2