Skip to content
This repository was archived by the owner on Jun 3, 2025. It is now read-only.

[Chat pipeline] session context manager#1276

Merged
dsikka merged 1 commit intomainfrom
chat-session-manager
Sep 25, 2023
Merged

[Chat pipeline] session context manager#1276
dsikka merged 1 commit intomainfrom
chat-session-manager

Conversation

@bfineran
Copy link
Copy Markdown
Contributor

updated implementation of #1239

includes small bug fix for text gen input parsing due to behavior with pydantic aliases

requested UX:

with chat_pipeline.session():
    first_result = chat_pipeline(prompt=PROMPT, *args, **kwargs)
    second_result = chat_pipeline(prompt=PROMPT, *args, **kwargs)

test_plan:
unit test included and ran locally, but disabled due to overhead in GHA

manual verification as well:

from deepsparse import Pipeline

pipeline = Pipeline.create(
    "chat",
    model_path="/home/benjamin/neuralmagic/models/codegen_mono-350m-bigpython_bigquery_thepile-pruned50/deployment",
    engine_type="onnxruntime"
)

with pipeline.session():
    output_1 = pipeline(prompt="first", generation_config=dict(max_new_tokens=1))
    output_2 = pipeline(prompt="second", generation_config=dict(max_new_tokens=1))

output_3 = pipeline(prompt="third", generation_config=dict(max_new_tokens=1))

output_2.session_ids == output_1.session_ids  # expected True
output_3.session_ids == output_1.session_ids  # expected False

Co-authored-by: @rahul-tuli

Co-authored-by: rahul-tuli
@dsikka dsikka merged commit d13cc2d into main Sep 25, 2023
@dsikka dsikka deleted the chat-session-manager branch September 25, 2023 19:06
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants