Feat/bedrock support for Nova models through the ConverseAPI #207
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MiNeves00
added
enhancement
diogoncalves
approved these changes
Feb 10, 2025
diogoncalves
added a commit
that referenced
this pull request
Feb 12, 2025
…LLMstudio-proxy 1.0.5 (#195) * chore: Provider Unit Tests (#173) * chore: added unit tests for core provider. small bugfix on calculate_metrics of provider * added unit tests and docstring for join chunks * added unit tests and docstrings for calculate_cost on provider * added unit tests and docstrings for input_to_string on provider * added unit tests and docstrings for chat and achat * added unit tests and docstrings for chat and achat * chore: cleaned provider unit tests * chore: separated provider tests into different files. fixed some of its tests * chore: linted code * chore: deleted some comments * chore: linted * chore: Added Azure Provider Unit Tests (#176) * chore: added unit tests for azure provider * chore: added more unit tests and docstrings on azure, removed redundant comments * chore: added unit tests for generate client on Azure Provider * chore: separated azure unit tests into separate files. fixed some of its tests. * chore: linted code * chore: new line Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Co-authored-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: rename action * feat: added action to run tests on PR * chore: comments * fix: fix azure config tests * chore: style format * fix: tests workflow * Feature/prompt management (#200) * [feat] prompt management * [feat] testing * [feat] only one active prompt * [fix] bump prerelease version in pyproject.toml * [bugfix] return empty prompt * [fix] bump prerelease version in pyproject.toml * Update CONTRIBUTING.md Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Feat/ Use Openai Usage to calculate Cache and Reasoning Costs (#199) * feat: collects usage from stream and non stream openai calls * chore: refactored to provider to have a Metrics obj * feat: calculate_metrics now takes into account cached & reasoning tokens. Prices of openai models updated * fix: added caching tokens to model config obj * chore: added integration test for cache and reasoning * chore: added integration test for usage retrieval when max tokens reached * chore: uncommented runs from examples/core.py * fix: bugfix regarding usage on function calling. added a test for this * chore: merged with develop * chore: extracted provider data structures to another file * chore: renamed to private methods some within provider. splitted integration tests into 2 files * chore: deletion of a todo comment * chore: update poetry.lock * chore: specify python versions * chore: moving langchain integration tests to sdk * chore: format * feat: added support for o3-mini and updated o1-mini prices. also updated integration tests to support o3 (#202) * chore: removed duplicated code; removed duplicated integration tests * chore: updated github actions to run integration tests * chore: fixing github actions * chore: fixing github actions again * chore: fixing github actions again-x2 * chore: fixing github actions again-x2 * chore: added cache of dependencies to integration-tests in githubaction * chore: updated integration-tests action to inject github secrets into env * Feat/bedrock support for Nova models through the ConverseAPI (#207) * feat: added support for bedrock nova models * feat: tokens are now read from usage if available to ensure accuracy * chore: removed duplicated integration tests folder in wrong place * feat: refactored bedrock provider into being a single file instead of folder * chore: renamed bedrock to bedrock-converse in examples/core.py * chore: renamed bedrock in config.yaml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * Update pyproject.toml updated llmstudio-tracker version Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: updated llmstudio sdk poetry.lock --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: GitHub Actions <actions@github.com> Co-authored-by: brunoalho99 <132477278+brunoalho99@users.noreply.github.com> Co-authored-by: brunoalho <bruno.alho@tensorops.ai> Co-authored-by: Miguel Neves <miguel.neves.filipe@gmail.com>
diogoncalves
added a commit
that referenced
this pull request
Apr 21, 2025
* chore: Provider Unit Tests (#173) * chore: added unit tests for core provider. small bugfix on calculate_metrics of provider * added unit tests and docstring for join chunks * added unit tests and docstrings for calculate_cost on provider * added unit tests and docstrings for input_to_string on provider * added unit tests and docstrings for chat and achat * added unit tests and docstrings for chat and achat * chore: cleaned provider unit tests * chore: separated provider tests into different files. fixed some of its tests * chore: linted code * chore: deleted some comments * chore: linted * chore: Added Azure Provider Unit Tests (#176) * chore: added unit tests for azure provider * chore: added more unit tests and docstrings on azure, removed redundant comments * chore: added unit tests for generate client on Azure Provider * chore: separated azure unit tests into separate files. fixed some of its tests. * chore: linted code * chore: new line Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Co-authored-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: rename action * feat: added action to run tests on PR * chore: comments * fix: fix azure config tests * chore: style format * fix: tests workflow * Feature/prompt management (#200) * [feat] prompt management * [feat] testing * [feat] only one active prompt * [fix] bump prerelease version in pyproject.toml * [bugfix] return empty prompt * [fix] bump prerelease version in pyproject.toml * Update CONTRIBUTING.md Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Feat/ Use Openai Usage to calculate Cache and Reasoning Costs (#199) * feat: collects usage from stream and non stream openai calls * chore: refactored to provider to have a Metrics obj * feat: calculate_metrics now takes into account cached & reasoning tokens. Prices of openai models updated * fix: added caching tokens to model config obj * chore: added integration test for cache and reasoning * chore: added integration test for usage retrieval when max tokens reached * chore: uncommented runs from examples/core.py * fix: bugfix regarding usage on function calling. added a test for this * chore: merged with develop * chore: extracted provider data structures to another file * chore: renamed to private methods some within provider. splitted integration tests into 2 files * chore: deletion of a todo comment * chore: update poetry.lock * chore: specify python versions * chore: moving langchain integration tests to sdk * chore: format * feat: added support for o3-mini and updated o1-mini prices. also updated integration tests to support o3 (#202) * chore: removed duplicated code; removed duplicated integration tests * chore: updated github actions to run integration tests * chore: fixing github actions * chore: fixing github actions again * chore: fixing github actions again-x2 * chore: fixing github actions again-x2 * chore: added cache of dependencies to integration-tests in githubaction * chore: updated integration-tests action to inject github secrets into env * Feat/bedrock support for Nova models through the ConverseAPI (#207) * feat: added support for bedrock nova models * feat: tokens are now read from usage if available to ensure accuracy * chore: removed duplicated integration tests folder in wrong place * feat: refactored bedrock provider into being a single file instead of folder * chore: renamed bedrock to bedrock-converse in examples/core.py * chore: renamed bedrock in config.yaml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * Update pyproject.toml updated llmstudio-tracker version Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: updated llmstudio sdk poetry.lock * Feat/converse support images (#211) * feat: added converse-api support for images in input. started making an integration test for this. * chore: added integration test for converse image sending * chore: send images integration test now also tests for openai * chore: integration test of send_imgs added async testing * chore: updated examples core.py to also have send images * feat: bedrock image input is now same contract as openai * chore: ChatCompletionLLMstudio print now hides large image bytes for readability * chore: fixes in the pretty print of ChatCompletionLLMstudio * chore: small fix in examples/core.py * fix: test_send_imgs had bug on reading env * chore: made clean_print optional on chatcompletions; image from url is directly converted to bytes * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * Update pyproject.toml Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Update pyproject.toml Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * chore: format --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: GitHub Actions <actions@github.com> Co-authored-by: brunoalho99 <132477278+brunoalho99@users.noreply.github.com> Co-authored-by: brunoalho <bruno.alho@tensorops.ai> Co-authored-by: Miguel Neves <miguel.neves.filipe@gmail.com>
diogoncalves
added a commit
that referenced
this pull request
Apr 22, 2025
* chore: Provider Unit Tests (#173) * chore: added unit tests for core provider. small bugfix on calculate_metrics of provider * added unit tests and docstring for join chunks * added unit tests and docstrings for calculate_cost on provider * added unit tests and docstrings for input_to_string on provider * added unit tests and docstrings for chat and achat * added unit tests and docstrings for chat and achat * chore: cleaned provider unit tests * chore: separated provider tests into different files. fixed some of its tests * chore: linted code * chore: deleted some comments * chore: linted * chore: Added Azure Provider Unit Tests (#176) * chore: added unit tests for azure provider * chore: added more unit tests and docstrings on azure, removed redundant comments * chore: added unit tests for generate client on Azure Provider * chore: separated azure unit tests into separate files. fixed some of its tests. * chore: linted code * chore: new line Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Co-authored-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: rename action * feat: added action to run tests on PR * chore: comments * fix: fix azure config tests * chore: style format * fix: tests workflow * Feature/prompt management (#200) * [feat] prompt management * [feat] testing * [feat] only one active prompt * [fix] bump prerelease version in pyproject.toml * [bugfix] return empty prompt * [fix] bump prerelease version in pyproject.toml * Update CONTRIBUTING.md Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Feat/ Use Openai Usage to calculate Cache and Reasoning Costs (#199) * feat: collects usage from stream and non stream openai calls * chore: refactored to provider to have a Metrics obj * feat: calculate_metrics now takes into account cached & reasoning tokens. Prices of openai models updated * fix: added caching tokens to model config obj * chore: added integration test for cache and reasoning * chore: added integration test for usage retrieval when max tokens reached * chore: uncommented runs from examples/core.py * fix: bugfix regarding usage on function calling. added a test for this * chore: merged with develop * chore: extracted provider data structures to another file * chore: renamed to private methods some within provider. splitted integration tests into 2 files * chore: deletion of a todo comment * chore: update poetry.lock * chore: specify python versions * chore: moving langchain integration tests to sdk * chore: format * feat: added support for o3-mini and updated o1-mini prices. also updated integration tests to support o3 (#202) * chore: removed duplicated code; removed duplicated integration tests * chore: updated github actions to run integration tests * chore: fixing github actions * chore: fixing github actions again * chore: fixing github actions again-x2 * chore: fixing github actions again-x2 * chore: added cache of dependencies to integration-tests in githubaction * chore: updated integration-tests action to inject github secrets into env * Feat/bedrock support for Nova models through the ConverseAPI (#207) * feat: added support for bedrock nova models * feat: tokens are now read from usage if available to ensure accuracy * chore: removed duplicated integration tests folder in wrong place * feat: refactored bedrock provider into being a single file instead of folder * chore: renamed bedrock to bedrock-converse in examples/core.py * chore: renamed bedrock in config.yaml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * Update pyproject.toml updated llmstudio-tracker version Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: updated llmstudio sdk poetry.lock * Feat/converse support images (#211) * feat: added converse-api support for images in input. started making an integration test for this. * chore: added integration test for converse image sending * chore: send images integration test now also tests for openai * chore: integration test of send_imgs added async testing * chore: updated examples core.py to also have send images * feat: bedrock image input is now same contract as openai * chore: ChatCompletionLLMstudio print now hides large image bytes for readability * chore: fixes in the pretty print of ChatCompletionLLMstudio * chore: small fix in examples/core.py * fix: test_send_imgs had bug on reading env * chore: made clean_print optional on chatcompletions; image from url is directly converted to bytes * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * feat: adapt langchain integration * chore: update lock * chore: make format --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: GitHub Actions <actions@github.com> Co-authored-by: brunoalho99 <132477278+brunoalho99@users.noreply.github.com> Co-authored-by: brunoalho <bruno.alho@tensorops.ai> Co-authored-by: Miguel Neves <miguel.neves.filipe@gmail.com>
diogoncalves
added a commit
that referenced
this pull request
Apr 24, 2025
* Feature/langraph integration (#215) * chore: Provider Unit Tests (#173) * chore: added unit tests for core provider. small bugfix on calculate_metrics of provider * added unit tests and docstring for join chunks * added unit tests and docstrings for calculate_cost on provider * added unit tests and docstrings for input_to_string on provider * added unit tests and docstrings for chat and achat * added unit tests and docstrings for chat and achat * chore: cleaned provider unit tests * chore: separated provider tests into different files. fixed some of its tests * chore: linted code * chore: deleted some comments * chore: linted * chore: Added Azure Provider Unit Tests (#176) * chore: added unit tests for azure provider * chore: added more unit tests and docstrings on azure, removed redundant comments * chore: added unit tests for generate client on Azure Provider * chore: separated azure unit tests into separate files. fixed some of its tests. * chore: linted code * chore: new line Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Co-authored-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: rename action * feat: added action to run tests on PR * chore: comments * fix: fix azure config tests * chore: style format * fix: tests workflow * Feature/prompt management (#200) * [feat] prompt management * [feat] testing * [feat] only one active prompt * [fix] bump prerelease version in pyproject.toml * [bugfix] return empty prompt * [fix] bump prerelease version in pyproject.toml * Update CONTRIBUTING.md Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Feat/ Use Openai Usage to calculate Cache and Reasoning Costs (#199) * feat: collects usage from stream and non stream openai calls * chore: refactored to provider to have a Metrics obj * feat: calculate_metrics now takes into account cached & reasoning tokens. Prices of openai models updated * fix: added caching tokens to model config obj * chore: added integration test for cache and reasoning * chore: added integration test for usage retrieval when max tokens reached * chore: uncommented runs from examples/core.py * fix: bugfix regarding usage on function calling. added a test for this * chore: merged with develop * chore: extracted provider data structures to another file * chore: renamed to private methods some within provider. splitted integration tests into 2 files * chore: deletion of a todo comment * chore: update poetry.lock * chore: specify python versions * chore: moving langchain integration tests to sdk * chore: format * feat: added support for o3-mini and updated o1-mini prices. also updated integration tests to support o3 (#202) * chore: removed duplicated code; removed duplicated integration tests * chore: updated github actions to run integration tests * chore: fixing github actions * chore: fixing github actions again * chore: fixing github actions again-x2 * chore: fixing github actions again-x2 * chore: added cache of dependencies to integration-tests in githubaction * chore: updated integration-tests action to inject github secrets into env * Feat/bedrock support for Nova models through the ConverseAPI (#207) * feat: added support for bedrock nova models * feat: tokens are now read from usage if available to ensure accuracy * chore: removed duplicated integration tests folder in wrong place * feat: refactored bedrock provider into being a single file instead of folder * chore: renamed bedrock to bedrock-converse in examples/core.py * chore: renamed bedrock in config.yaml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * Update pyproject.toml updated llmstudio-tracker version Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> * [fix] bump prerelease version in pyproject.toml * chore: updated llmstudio sdk poetry.lock * Feat/converse support images (#211) * feat: added converse-api support for images in input. started making an integration test for this. * chore: added integration test for converse image sending * chore: send images integration test now also tests for openai * chore: integration test of send_imgs added async testing * chore: updated examples core.py to also have send images * feat: bedrock image input is now same contract as openai * chore: ChatCompletionLLMstudio print now hides large image bytes for readability * chore: fixes in the pretty print of ChatCompletionLLMstudio * chore: small fix in examples/core.py * fix: test_send_imgs had bug on reading env * chore: made clean_print optional on chatcompletions; image from url is directly converted to bytes * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * feat: adapt langchain integration * chore: update lock * chore: make format --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: GitHub Actions <actions@github.com> Co-authored-by: brunoalho99 <132477278+brunoalho99@users.noreply.github.com> Co-authored-by: brunoalho <bruno.alho@tensorops.ai> Co-authored-by: Miguel Neves <miguel.neves.filipe@gmail.com> * [fix] bump prerelease version in pyproject.toml * [fix] bump prerelease version in pyproject.toml * Update core.py Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Update core.py Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Update core.py Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> * Update core.py Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> --------- Signed-off-by: Diogo Goncalves <diogoncalves@users.noreply.github.com> Signed-off-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: Miguel Neves <61327611+MiNeves00@users.noreply.github.com> Co-authored-by: GitHub Actions <actions@github.com> Co-authored-by: brunoalho99 <132477278+brunoalho99@users.noreply.github.com> Co-authored-by: brunoalho <bruno.alho@tensorops.ai> Co-authored-by: Miguel Neves <miguel.neves.filipe@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LLMstudio Version X.X.X
What was done in this PR:
How it was tested:
python examples/core.pywhere a new example for a nova model was added and ran successfully (tested with all 3 added models).Additional notes:
Due to these Nova models currently using cross-region for inference, had to add the prefix
us.to the model name (ex:us.amazon.nova-pro-v1:0). Currently the model is only available in the US so its okay.us.toeu.) or for us to add multiple entries in the config.yaml (although it would make it harder to manage and keep all the prices updated).Token counting now prioritizes info from the Usage if available instead of just trying to count using a tokenizer.
Maybe we should consider creating a Usage Class obj in the future for better clarity and maintanability
No breaking changes should happen.