Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function Calling Support for Gemini - Part 2 #3726

Merged
merged 73 commits into from
Oct 10, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
ec8d14f
Added function calling support to GeminiClient
arjun-g May 25, 2024
4e7aa8b
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g May 26, 2024
bf70633
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g May 26, 2024
2d7a5df
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g May 27, 2024
eafd0d7
Merge branch 'main' into dev
BeibinLi May 28, 2024
4e8a736
Appending a continue message from model to alternate between user and…
arjun-g May 28, 2024
592e877
Merge branch 'dev' of https://github.com/arjun-g/autogen into dev
arjun-g May 28, 2024
c4a9c10
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g May 29, 2024
a9947ad
Fixed cost calculation to include <128K costing and new 1.5-flash mod…
arjun-g May 29, 2024
0ce915c
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g May 29, 2024
d3301b1
Add notebook with samples for Gemini
arjun-g Jun 2, 2024
5902f9e
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g Jun 2, 2024
4812abc
Merge branch 'main' into dev
BeibinLi Jun 3, 2024
323f2fd
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g Jun 4, 2024
3a9e9d1
Merge branch 'dev' of https://github.com/arjun-g/autogen into dev
arjun-g Jun 4, 2024
d01e554
Updated test case
arjun-g Jun 4, 2024
ae2641f
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g Jun 4, 2024
668f553
Fix to handle not dict response in function call
arjun-g Jun 18, 2024
5cb7c9d
Handle non dict function results and append dummy model message betwe…
arjun-g Jun 18, 2024
187702d
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g Jun 18, 2024
6343132
Fixing message order in gemini
arjun-g Jun 19, 2024
614090e
Merge branch 'main' of https://github.com/microsoft/autogen into dev
arjun-g Jun 19, 2024
ec69cd6
Append text as multiple parts instead of test concatenation
arjun-g Jun 19, 2024
eba86c3
Merge branch 'main' into dev
arjun-g Jun 20, 2024
d64d016
Raising error for Union data types in function parameter
arjun-g Jun 20, 2024
0223b60
Merge branch 'dev' of https://github.com/arjun-g/autogen into dev
arjun-g Jun 20, 2024
f9ab9b8
Delete default key
arjun-g Jun 20, 2024
c827205
Merge branch 'main' into dev
arjun-g Jun 20, 2024
d69a4ff
Merge branch 'main' into dev
arjun-g Jun 21, 2024
f0880f2
Merge branch 'main' into dev
sonichi Jun 21, 2024
3504ee7
Update gemini.py for multiple tool calls + pre-commit formatting
marklysze Jun 25, 2024
6563248
Merge branch 'main' into dev
marklysze Jun 25, 2024
c73c5a0
Merge branch 'main' into dev
luxzoli Aug 9, 2024
498d2aa
Merge branch 'microsoft:main' into arjun-g-dev
luxzoli Aug 9, 2024
3b8efee
no function role
luxzoli Aug 9, 2024
38d533a
Merge branch 'microsoft:main' into arjun-g-dev
luxzoli Aug 12, 2024
b06236c
Merge branch 'microsoft:main' into arjun-g-dev
luxzoli Aug 12, 2024
b8faaea
Merge branch 'microsoft:main' into arjun-g-dev
luxzoli Aug 14, 2024
9efbc5d
Merge branch 'microsoft:main' into arjun-g-dev
luxzoli Aug 16, 2024
076ccd9
Merge branch 'microsoft:main' into arjun-g-dev
luxzoli Aug 17, 2024
b4bd39c
Merge branch 'microsoft:main' into arjun-g-dev
luxzoli Aug 18, 2024
a49690e
start adding function calling config
luxzoli Aug 18, 2024
f79d695
do not serialize tool_config
luxzoli Aug 18, 2024
90fbd41
improve tool config parsing
luxzoli Aug 18, 2024
409774a
add hint
luxzoli Aug 18, 2024
26dd563
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 19, 2024
511b5ec
improve function calling config
luxzoli Aug 19, 2024
86e56ff
removunnecessary comments
luxzoli Aug 19, 2024
d563217
try removing allowed function names in tool config conversion
luxzoli Aug 19, 2024
cd6e447
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 20, 2024
8f8ec88
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 20, 2024
76622a2
fix tool config parsing with empty tools list
luxzoli Aug 20, 2024
ae49a3a
improve logging and case handling with vertexai tool config parsing
luxzoli Aug 20, 2024
83ba5c1
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 21, 2024
a3388c5
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 22, 2024
060f2aa
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 26, 2024
519f7d0
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 27, 2024
d34ea0d
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 27, 2024
93bc0b9
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 28, 2024
2c26da8
reset file
luxzoli Aug 28, 2024
144d9ad
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Aug 28, 2024
3f0fea6
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Sep 2, 2024
7e7b6db
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Sep 6, 2024
bf56d77
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Sep 10, 2024
b2971e9
check if text is in part
luxzoli Sep 23, 2024
64af078
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Sep 25, 2024
1a3f618
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Oct 3, 2024
dcc827d
fix empty part checking case
luxzoli Oct 7, 2024
e82ab71
Merge branch 'microsoft:main' into arjun-g-dev-tool-config
luxzoli Oct 7, 2024
99fb7f7
fix bug with attribute handling
luxzoli Oct 9, 2024
17820e1
Merge branch '0.2' into arjun-g-dev-tool-config
luxzoli Oct 9, 2024
8f9caf6
Merge branch '0.2' into arjun-g-dev-tool-config
luxzoli Oct 10, 2024
d1391e4
skip test if gemini deps are not installed
luxzoli Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix tool config parsing with empty tools list
  • Loading branch information
luxzoli committed Aug 20, 2024
commit 76622a2b67725473c8d92076ce2341c30040e407
12 changes: 8 additions & 4 deletions autogen/oai/gemini.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ def __init__(self, **kwargs):

Args:
api_key (str): The API key for using Gemini.
credentials (google.auth.credentials.Credentials): credentials to be used for authentication with vertexai.
credentials (google.auth.credentials.Credentials): credentials to be used for authentication with vertexai.
google_application_credentials (str): Path to the JSON service account key file of the service account.
Alternatively, the GOOGLE_APPLICATION_CREDENTIALS environment variable
can also be set instead of using this argument.
Expand Down Expand Up @@ -647,9 +647,13 @@ def _to_vertexai_tool_config(tool_config, tools):
func_calling_config_params["mode"] = VertexAIToolConfig.FunctionCallingConfig.Mode[
tool_config["function_calling_config"]["mode"]
]
# if func_calling_config_params["mode"] == VertexAIToolConfig.FunctionCallingConfig.Mode.ANY:
# # The function names are not yet known when parsing the OAI_CONFIG_LIST
# func_calling_config_params["allowed_function_names"] = [tool["function_name"] for tool in tools]
if (
(func_calling_config_params["mode"] == VertexAIToolConfig.FunctionCallingConfig.Mode.ANY)
and (len(tools) > 0)
and all(["function_name" in tool for tool in tools])
):
# The function names are not yet known when parsing the OAI_CONFIG_LIST
func_calling_config_params["allowed_function_names"] = [tool["function_name"] for tool in tools]
vertexai_tool_config = VertexAIToolConfig(
function_calling_config=VertexAIToolConfig.FunctionCallingConfig(**func_calling_config_params)
)
Expand Down
26 changes: 26 additions & 0 deletions test/oai/test_gemini.py
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,32 @@ def test_vertexai_tool_config(gemini_client):
), "Function calling allowed function names is not converted correctly"


@pytest.mark.skipif(skip, reason="Google GenAI dependency is not installed")
def test_vertexai_tool_config_no_functions(gemini_client):

tools = []

tool_config = {"function_calling_config": {"mode": "ANY"}}

expected_tool_config = VertexAIToolConfig(
function_calling_config=VertexAIToolConfig.FunctionCallingConfig(
mode=VertexAIToolConfig.FunctionCallingConfig.Mode.ANY,
)
)

converted_tool_config = GeminiClient._to_vertexai_tool_config(tool_config, tools)

converted_mode = converted_tool_config._gapic_tool_config.function_calling_config.mode
expected_mode = expected_tool_config._gapic_tool_config.function_calling_config.mode
converted_allowed_func = converted_tool_config._gapic_tool_config.function_calling_config.allowed_function_names
expected_allowed_func = expected_tool_config._gapic_tool_config.function_calling_config.allowed_function_names

assert converted_mode == expected_mode, "Function calling mode is not converted correctly"
assert (
converted_allowed_func == expected_allowed_func
), "Function calling allowed function names is not converted correctly"


# Test error handling
@patch("autogen.oai.gemini.genai")
@pytest.mark.skipif(skip, reason="Google GenAI dependency is not installed")
Expand Down