Add json schema #7
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
Rolland-He
reviewed
Jul 10, 2025
Collaborator
Rolland-He
left a comment
Rolland-He
left a comment
There was a problem hiding this comment.
Hi, @wkukka1 nice work! I’ve left a inline comment regarding magic strings, mostly suggesting to extract repeated literals into named constants. Just a minor style suggestion and totally fine to leave as is. Everything else looks good!
ai_server/server.py
Outdated
| messages = _build_messages(content, system_prompt, image_files=[]) # TODO: Pass image files | ||
| payload = {'model': model, 'messages': messages, 'stream': False, 'max_tokens': 512} | ||
| if json_schema: | ||
| payload['json_schema'] = json_schema["schema"] |
Collaborator
There was a problem hiding this comment.
For coding style, try to define "schema" as a constant like SCHEMA_KEY = "schema" and reuse it throughout. This makes the intent clearer and avoids potential typos. Similarly for "--json-schema".
ai_server/server.py
Outdated
| model=model, | ||
| messages=messages, | ||
| stream=False, | ||
| format=json_schema['schema'] if json_schema else None, |
| cmd = [LLAMA_CPP_CLI, '-m', model_path, '--n-gpu-layers', '40', '-p', content, '-n', '512', '--single-turn'] | ||
| if json_schema: | ||
| raw_schema = json_schema["schema"] if "schema" in json_schema else json_schema | ||
| cmd += ["--json-schema", json.dumps(raw_schema)] |
for more information, see https://pre-commit.ci
Rolland-He
reviewed
Jul 10, 2025
Collaborator
Rolland-He
left a comment
Rolland-He
left a comment
There was a problem hiding this comment.
Hi @wkukka1 , good work! There's still one place left.
ai_server/server.py
Outdated
| model=model, | ||
| messages=messages, | ||
| stream=False, | ||
| format=json_schema['schema'] if json_schema else None, |
added 2 commits
July 10, 2025 15:03
test/test_server_mode.py
Outdated
| assert args[0] == "http://localhost:8080/v1/chat/completions" | ||
|
|
||
| body = kwargs["json"] | ||
| print(body) |
david-yz-liu
approved these changes
Jul 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates:
json_schemaparam in post request to/chatendpoint