Skip to content

Commit

Permalink
Python: Google AI Connector (#7419)
Browse files Browse the repository at this point in the history
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
We are adding a new AI connector for the talking to the Gemini API on
Google AI:
https://ai.google.dev/gemini-api/docs/get-started/tutorial?authuser=1&lang=python.

Note that this is for Gemini hosted on Google AI. Google also offers
Gemini access on their Vertex AI platform:
https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/migrate-google-ai.
We will have another connector along with this for accessing Gemini on
Vertex AI.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

The new connector contains 3 AI services:
- Chat completion
- Text completion
- Text embedding

TODO: Function calling.
> Function calling is not included in this PR to reduce to the size of
the PR.

Unit test coverage:

![image](https://github.com/user-attachments/assets/d4fefaa3-b24a-4b91-b29a-cadec5fa34fb)

Mypy: All cleared

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
  • Loading branch information
TaoChenOSU authored Jul 24, 2024
1 parent 57c7253 commit d162147
Show file tree
Hide file tree
Showing 27 changed files with 1,889 additions and 17 deletions.
6 changes: 6 additions & 0 deletions .github/workflows/python-integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,9 @@ jobs:
MISTRALAI_API_KEY: ${{secrets.MISTRALAI_API_KEY}}
MISTRALAI_CHAT_MODEL_ID: ${{ vars.MISTRALAI_CHAT_MODEL_ID }}
OLLAMA_MODEL: "${{ matrix.os == 'ubuntu-latest' && vars.OLLAMA_MODEL || '' }}" # phi3
GOOGLE_AI_GEMINI_MODEL_ID: ${{ vars.GOOGLE_AI_GEMINI_MODEL_ID }}
GOOGLE_AI_EMBEDDING_MODEL_ID: ${{ vars.GOOGLE_AI_EMBEDDING_MODEL_ID }}
GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_AI_API_KEY }}
run: |
if ${{ matrix.os == 'ubuntu-latest' }}; then
docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest
Expand Down Expand Up @@ -193,6 +196,9 @@ jobs:
MISTRALAI_API_KEY: ${{secrets.MISTRALAI_API_KEY}}
MISTRALAI_CHAT_MODEL_ID: ${{ vars.MISTRALAI_CHAT_MODEL_ID }}
OLLAMA_MODEL: "${{ matrix.os == 'ubuntu-latest' && vars.OLLAMA_MODEL || '' }}" # phi3
GOOGLE_AI_GEMINI_MODEL_ID: ${{ vars.GOOGLE_AI_GEMINI_MODEL_ID }}
GOOGLE_AI_EMBEDDING_MODEL_ID: ${{ vars.GOOGLE_AI_EMBEDDING_MODEL_ID }}
GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_AI_API_KEY }}
run: |
if ${{ matrix.os == 'ubuntu-latest' }}; then
docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest
Expand Down
6 changes: 5 additions & 1 deletion python/.cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@
"kernelfunction",
"skprompt",
"templating",
"vectordb"
"vectordb",
"generativeai",
"genai",
"protos",
"endregion"
]
}
181 changes: 178 additions & 3 deletions python/poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ azure-identity = { version = "^1.13.0", optional = true}
azure-cosmos = { version = "^4.7.0", optional = true}
# chroma
chromadb = { version = ">=0.4.13,<0.6.0", optional = true}
# google
google-generativeai = { version = "^0.7.2", optional = true}
# hugging face
transformers = { version = "^4.28.1", extras=["torch"], optional = true}
sentence-transformers = { version = "^2.2.2", optional = true}
Expand Down Expand Up @@ -92,6 +94,7 @@ azure-core = "^1.28.0"
azure-cosmos = "^4.7.0"
mistralai = "^0.4.1"
ollama = "^0.2.1"
google-generativeai = "^0.7.2"
transformers = { version = "^4.28.1", extras=["torch"]}
sentence-transformers = "^2.2.2"

Expand All @@ -108,6 +111,8 @@ azure-cosmos = "^4.7.0"
msgraph-sdk = "^1.2.0"
# chroma
chromadb = ">=0.4.13,<0.6.0"
# google
google-generativeai = "^0.7.2"
# hugging face
transformers = { version = "^4.28.1", extras=["torch"]}
sentence-transformers = "^2.2.2"
Expand Down Expand Up @@ -136,10 +141,11 @@ weaviate-client = ">=3.18,<5.0"

# Extras are exposed to pip, this allows a user to easily add the right dependencies to their environment
[tool.poetry.extras]
all = ["transformers", "sentence-transformers", "qdrant-client", "chromadb", "pymilvus", "milvus", "mistralai", "ollama", "weaviate-client", "pinecone-client", "psycopg", "redis", "azure-ai-inference", "azure-search-documents", "azure-core", "azure-identity", "azure-cosmos", "usearch", "pyarrow", "ipykernel", "motor"]
all = ["transformers", "sentence-transformers", "qdrant-client", "chromadb", "pymilvus", "milvus", "mistralai", "ollama", "google", "weaviate-client", "pinecone-client", "psycopg", "redis", "azure-ai-inference", "azure-search-documents", "azure-core", "azure-identity", "azure-cosmos", "usearch", "pyarrow", "ipykernel", "motor"]

azure = ["azure-ai-inference", "azure-search-documents", "azure-core", "azure-identity", "azure-cosmos", "msgraph-sdk"]
chromadb = ["chromadb"]
google = ["google-generativeai"]
hugging_face = ["transformers", "sentence-transformers"]
milvus = ["pymilvus", "milvus"]
mistralai = ["mistralai"]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Copyright (c) Microsoft. All rights reserved.

import sys
from typing import TYPE_CHECKING, Any

from azure.ai.inference.aio import EmbeddingsClient
Expand All @@ -8,6 +9,11 @@
from numpy import array, ndarray
from pydantic import ValidationError

if sys.version_info >= (3, 12):
from typing import override # pragma: no cover
else:
from typing_extensions import override # pragma: no cover

from semantic_kernel.connectors.ai.azure_ai_inference.azure_ai_inference_prompt_execution_settings import (
AzureAIInferenceEmbeddingPromptExecutionSettings,
)
Expand Down Expand Up @@ -98,7 +104,13 @@ async def generate_embeddings(
dimensions=settings.dimensions if settings else None,
encoding_format=settings.encoding_format if settings else None,
input_type=settings.input_type if settings else None,
kwargs=kwargs,
)

return array([array(item.embedding) for item in response.data])

@override
def get_prompt_execution_settings_class(
self,
) -> type["PromptExecutionSettings"]:
"""Get the request settings class."""
return AzureAIInferenceEmbeddingPromptExecutionSettings
Loading

0 comments on commit d162147

Please sign in to comment.