Python: Introduce Pydantic settings (microsoft#6193)

### Motivation and Context

SK Python is tightly coupled to the use of a `.env` file to read all
secrets, keys, endpoints, and more. This doesn't scale well for users
who wish to be able to use environment variables with their SK
Applications. By introducing Pydantic Settings, it is possible to use
both environment variables as well as have a fall-back to a `.env` file
(via a `env_file_path` parameter), if desired.

By introducing Pydantic Settings, we are removing the requirement to
have to create Text/Embedding/Chat completion objects with an `api_key`
or other previously required information (in the case of
AzureChatCompletion that means an `endpoint`, an `api_key`, a
`deployment_name`, and an `api_version`). When the AI connector is
created, the Pydantic settings are loaded either via env vars or the
fall-back `.env` file, and that means the user can create a chat
completion object like:

```python
chat_completion = OpenAIChatCompletion(service_id="test")
```

or, to optionally override the `ai_model_id` env var:

```python
chat_completion = OpenAIChatCompletion(service_id="test", ai_model_id="gpt-4-1106")
```
Note: we have left the ability to specific an `api_key`/`org_id` for
`OpenAIChatCompletion` or a `deployment_name`, `endpoint`, `base_url`,
and `api_version` for `AzureChatCompletion` as before, but if your
settings are configured to use env vars/.env file then there is no need
to pass this information.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

The PR introduces the use of Pydantic settings and removes the use of
the python-dotenv library.
- Closes microsoft#1779 
- Updates notebooks, samples, code and tests to remove the explicit
config of api_key or other previous .env files values.
- Adds new unit test config using monkeypatch to simulate env variables
for testing
- All unit and integration tests passing

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄

Author: moonbox3

.github/workflows/python-integration-tests.yml

@@ -76,25 +76,21 @@ jobs:
         env: # Set Azure credentials secret as an input
           HNSWLIB_NO_NATIVE: 1
           Python_Integration_Tests: Python_Integration_Tests
-          AzureOpenAI__Label: azure-text-davinci-003
-          AzureOpenAIEmbedding__Label: azure-text-embedding-ada-002
-          AzureOpenAI__DeploymentName: ${{ vars.AZUREOPENAI__DEPLOYMENTNAME }}
-          AzureOpenAI__Text__DeploymentName: ${{ vars.AZUREOPENAI__TEXT__DEPLOYMENTNAME }}
-          AzureOpenAIChat__DeploymentName: ${{ vars.AZUREOPENAI__CHAT__DEPLOYMENTNAME }}
-          AzureOpenAIEmbeddings__DeploymentName: ${{ vars.AZUREOPENAIEMBEDDINGS__DEPLOYMENTNAME2 }}
-          AzureOpenAIEmbeddings_EastUS__DeploymentName: ${{ vars.AZUREOPENAIEMBEDDINGS_EASTUS__DEPLOYMENTNAME}}
-          AzureOpenAI__Endpoint: ${{ secrets.AZUREOPENAI__ENDPOINT }}
-          AzureOpenAI_EastUS__Endpoint: ${{ secrets.AZUREOPENAI_EASTUS__ENDPOINT }}
-          AzureOpenAI_EastUS__ApiKey: ${{ secrets.AZUREOPENAI_EASTUS__APIKEY }}
-          AzureOpenAIEmbeddings__Endpoint: ${{ secrets.AZUREOPENAI__ENDPOINT }}
-          AzureOpenAI__ApiKey: ${{ secrets.AZUREOPENAI__APIKEY }}
-          AzureOpenAIEmbeddings__ApiKey: ${{ secrets.AZUREOPENAI__APIKEY }}
-          Bing__ApiKey: ${{ secrets.BING__APIKEY }}
-          OpenAI__ApiKey: ${{ secrets.OPENAI__APIKEY }}
-          Pinecone__ApiKey: ${{ secrets.PINECONE__APIKEY }}
-          Postgres__Connectionstr: ${{secrets.POSTGRES__CONNECTIONSTR}}
-          AZURE_COGNITIVE_SEARCH_ADMIN_KEY: ${{secrets.AZURE_COGNITIVE_SEARCH_ADMIN_KEY}}
-          AZURE_COGNITIVE_SEARCH_ENDPOINT: ${{secrets.AZURE_COGNITIVE_SEARCH_ENDPOINT}}
+          AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME }} # azure-text-embedding-ada-002
+          AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
+          AZURE_OPENAI_TEXT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_TEXT_DEPLOYMENT_NAME }}
+          AZURE_OPENAI_API_VERSION: ${{ vars.AZURE_OPENAI_API_VERSION }}
+          AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
+          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
+          BING_API_KEY: ${{ secrets.BING_API_KEY }}
+          OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_ID }}
+          OPENAI_TEXT_MODEL_ID: ${{ vars.OPENAI_TEXT_MODEL_ID }}
+          OPENAI_EMBEDDING_MODEL_ID: ${{ vars.OPENAI_EMBEDDING_MODEL_ID }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          PINECONE_API_KEY: ${{ secrets.PINECONE__APIKEY }}
+          POSTGRES_CONNECTION_STRING: ${{secrets.POSTGRES__CONNECTIONSTR}}
+          AZURE_AI_SEARCH_API_KEY: ${{secrets.AZURE_AI_SEARCH_API_KEY}}
+          AZURE_AI_SEARCH_ENDPOINT: ${{secrets.AZURE_AI_SEARCH_ENDPOINT}}
           MONGODB_ATLAS_CONNECTION_STRING: ${{secrets.MONGODB_ATLAS_CONNECTION_STRING}}
         run: |
           if ${{ matrix.os == 'ubuntu-latest' }}; then
@@ -142,25 +138,21 @@ jobs:
         env: # Set Azure credentials secret as an input
           HNSWLIB_NO_NATIVE: 1
           Python_Integration_Tests: Python_Integration_Tests
-          AzureOpenAI__Label: azure-text-davinci-003
-          AzureOpenAIEmbedding__Label: azure-text-embedding-ada-002
-          AzureOpenAI__DeploymentName: ${{ vars.AZUREOPENAI__DEPLOYMENTNAME }}
-          AzureOpenAI__Text__DeploymentName: ${{ vars.AZUREOPENAI__TEXT__DEPLOYMENTNAME }}
-          AzureOpenAIChat__DeploymentName: ${{ vars.AZUREOPENAI__CHAT__DEPLOYMENTNAME }}
-          AzureOpenAIEmbeddings__DeploymentName: ${{ vars.AZUREOPENAIEMBEDDINGS__DEPLOYMENTNAME2 }}
-          AzureOpenAIEmbeddings_EastUS__DeploymentName: ${{ vars.AZUREOPENAIEMBEDDINGS_EASTUS__DEPLOYMENTNAME}}
-          AzureOpenAI__Endpoint: ${{ secrets.AZUREOPENAI__ENDPOINT }}
-          AzureOpenAIEmbeddings__Endpoint: ${{ secrets.AZUREOPENAI__ENDPOINT }}
-          AzureOpenAI__ApiKey: ${{ secrets.AZUREOPENAI__APIKEY }}
-          AzureOpenAI_EastUS__Endpoint: ${{ secrets.AZUREOPENAI_EASTUS__ENDPOINT }}
-          AzureOpenAI_EastUS__ApiKey: ${{ secrets.AZUREOPENAI_EASTUS__APIKEY }}
-          AzureOpenAIEmbeddings__ApiKey: ${{ secrets.AZUREOPENAI__APIKEY }}
-          Bing__ApiKey: ${{ secrets.BING__APIKEY }}
-          OpenAI__ApiKey: ${{ secrets.OPENAI__APIKEY }}
-          Pinecone__ApiKey: ${{ secrets.PINECONE__APIKEY }}
-          Postgres__Connectionstr: ${{secrets.POSTGRES__CONNECTIONSTR}}
-          AZURE_COGNITIVE_SEARCH_ADMIN_KEY: ${{secrets.AZURE_COGNITIVE_SEARCH_ADMIN_KEY}}
-          AZURE_COGNITIVE_SEARCH_ENDPOINT: ${{secrets.AZURE_COGNITIVE_SEARCH_ENDPOINT}}
+          AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME }} # azure-text-embedding-ada-002
+          AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
+          AZURE_OPENAI_TEXT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_TEXT_DEPLOYMENT_NAME }}
+          AZURE_OPENAI_API_VERSION: ${{ vars.AZURE_OPENAI_API_VERSION }}
+          AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
+          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
+          BING_API_KEY: ${{ secrets.BING_API_KEY }}
+          OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_ID }}
+          OPENAI_TEXT_MODEL_ID: ${{ vars.OPENAI_TEXT_MODEL_ID }}
+          OPENAI_EMBEDDING_MODEL_ID: ${{ vars.OPENAI_EMBEDDING_MODEL_ID }}
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          PINECONE_API_KEY: ${{ secrets.PINECONE__APIKEY }}
+          POSTGRES_CONNECTION_STRING: ${{secrets.POSTGRES__CONNECTIONSTR}}
+          AZURE_AI_SEARCH_API_KEY: ${{secrets.AZURE_AI_SEARCH_API_KEY}}
+          AZURE_AI_SEARCH_ENDPOINT: ${{secrets.AZURE_AI_SEARCH_ENDPOINT}}
           MONGODB_ATLAS_CONNECTION_STRING: ${{secrets.MONGODB_ATLAS_CONNECTION_STRING}}
         run: |
           if ${{ matrix.os == 'ubuntu-latest' }}; then

python/.env.example

@@ -46,4 +46,9 @@ ASTRADB_APP_TOKEN=""
 ASTRADB_ID=""
 ASTRADB_REGION=""
 ASTRADB_KEYSPACE=""
-ACA_POOL_MANAGEMENT_ENDPOINT=""
+ACA_POOL_MANAGEMENT_ENDPOINT=""
+BOOKING_SAMPLE_CLIENT_ID=""
+BOOKING_SAMPLE_TENANT_ID=""
+BOOKING_SAMPLE_CLIENT_SECRET=""
+BOOKING_SAMPLE_BUSINESS_ID=""
+BOOKING_SAMPLE_SERVICE_ID=""

python/DEV_SETUP.md

@@ -10,17 +10,31 @@ Make sure you have an
 [OpenAI API Key](https://platform.openai.com) or
 [Azure OpenAI service key](https://learn.microsoft.com/azure/cognitive-services/openai/quickstart?pivots=rest-api)
 
-Copy those keys into a `.env` file (see the `.env.example` file):
+There are two methods to manage keys, secrets, and endpoints:
 
-```bash
+1. Store them in environment variables. SK Python leverages pydantic settings to load keys, secrets, and endpoints. This means that there is a first attempt to load them from environment variables. The `.env` file naming applies to how the names should be stored as environment variables.
+
+2. If you'd like to use the `.env` file, you will need to configure the `.env` file with the following keys into a `.env` file (see the `.env.example` file):
+
+```
 OPENAI_API_KEY=""
 OPENAI_ORG_ID=""
-AZURE_OPENAI_DEPLOYMENT_NAME=""
+AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=""
+AZURE_OPENAI_TEXT_DEPLOYMENT_NAME=""
+AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME=""
 AZURE_OPENAI_ENDPOINT=""
 AZURE_OPENAI_API_KEY=""
 ```
 
-We suggest adding a copy of the `.env` file under these folders:
+You will then configure the Text/ChatCompletion class with the keyword argument `env_file_path`:
+
+```python
+chat_completion = OpenAIChatCompletion(service_id="test", env_file_path=<path_to_file>)
+```
+
+This optional `env_file_path` parameter will allow pydantic settings to use the `.env` file as a fallback to read the settings.
+
+If using the second method, we suggest adding a copy of the `.env` file under these folders:
 
 - [python/tests](tests)
 - [./samples/getting_started](./samples/getting_started).

python/README.md

@@ -20,47 +20,52 @@ Make sure you have an
 [OpenAI API Key](https://platform.openai.com) or
 [Azure OpenAI service key](https://learn.microsoft.com/azure/cognitive-services/openai/quickstart?pivots=rest-api)
 
-Copy those keys into a `.env` file (see the `.env.example` file):
+There are two methods to manage keys, secrets, and endpoints:
+
+1. Store them in environment variables. SK Python leverages pydantic settings to load keys, secrets, and endpoints. This means that there is a first attempt to load them from environment variables. The `.env` file naming applies to how the names should be stored as environment variables.
+
+2. If you'd like to use the `.env` file, you will need to configure the `.env` file with the following keys in the file (see the `.env.example` file):
 
 ```
 OPENAI_API_KEY=""
 OPENAI_ORG_ID=""
-AZURE_OPENAI_DEPLOYMENT_NAME=""
+AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=""
+AZURE_OPENAI_TEXT_DEPLOYMENT_NAME=""
+AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME=""
 AZURE_OPENAI_ENDPOINT=""
 AZURE_OPENAI_API_KEY=""
 ```
 
+You will then configure the Text/ChatCompletion class with the keyword argument `env_file_path`:
+
+```python
+chat_completion = OpenAIChatCompletion(service_id="test", env_file_path=<path_to_file>)
+```
+
+This optional `env_file_path` parameter will allow pydantic settings to use the `.env` file as a fallback to read the settings.
+
 # Running a prompt
 
 ```python
 import asyncio
 from semantic_kernel import Kernel
 from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion, AzureChatCompletion
 from semantic_kernel.prompt_template import PromptTemplateConfig
-from semantic_kernel.utils.settings import openai_settings_from_dot_env, azure_openai_settings_from_dot_env
 
 kernel = Kernel()
 
 # Prepare OpenAI service using credentials stored in the `.env` file
-api_key, org_id = openai_settings_from_dot_env()
 service_id="chat-gpt"
 kernel.add_service(
     OpenAIChatCompletion(
         service_id=service_id,
-        ai_model_id="gpt-3.5-turbo",
-        api_key=api_key,
-        org_id=org_id
     )
 )
 
 # Alternative using Azure:
-# deployment, api_key, endpoint = azure_openai_settings_from_dot_env()
 # kernel.add_service(
 #   AzureChatCompletion(
 #       service_id=service_id,
-#       deployment_name=deployment,
-#       endpoint=endpoint,
-#       api_key=api_key
 #   )
 # )
 
@@ -112,10 +117,10 @@ if __name__ == "__main__":
 ```python
 # Create a reusable function summarize function
 summarize = kernel.add_function(
-        function_name="tldr_function",
-        plugin_name="tldr_plugin",
-        prompt="{{$input}}\n\nOne line TLDR with the fewest words.",
-        prompt_template_settings=req_settings,
+    function_name="tldr_function",
+    plugin_name="tldr_plugin",
+    prompt="{{$input}}\n\nOne line TLDR with the fewest words.",
+    prompt_template_settings=req_settings,
 )
 
 # Summarize the laws of thermodynamics

python/poetry.lock

@@ -24,11 +24,11 @@ grpcio = [
     { version = ">=1.60.0", python = ">=3.12" }
 ]
 openai = ">=1.0"
-python-dotenv = "^1.0.1"
 regex = "^2023.6.3"
 openapi_core = ">=0.18,<0.20"
 prance = "^23.6.21.0"
 pydantic = "^2"
+pydantic-settings = "^2.2.1"
 motor = "^3.3.2"
 defusedxml = "^0.7.1"
 pybars4 = "^0.9.13"

python/pyproject.toml

@@ -20,10 +20,6 @@
 from semantic_kernel.exceptions.function_exceptions import FunctionExecutionException
 from semantic_kernel.functions.kernel_arguments import KernelArguments
 from semantic_kernel.kernel import Kernel
-from semantic_kernel.utils.settings import (
-    azure_container_apps_settings_from_dot_env_as_dict,
-    azure_openai_settings_from_dot_env_as_dict,
-)
 
 auth_token: AccessToken | None = None
 
@@ -56,12 +52,11 @@ async def auth_callback() -> str:
 
 service_id = "sessions-tool"
 chat_service = AzureChatCompletion(
-    service_id=service_id, **azure_openai_settings_from_dot_env_as_dict(include_api_version=True)
+    service_id=service_id,
 )
 kernel.add_service(chat_service)
 
 sessions_tool = SessionsPythonTool(
-    **azure_container_apps_settings_from_dot_env_as_dict(),
     auth_callback=auth_callback,
 )
 

python/samples/concepts/auto_function_calling/azure_python_code_interpreter_function_calling.py

@@ -17,7 +17,6 @@
 from semantic_kernel.contents.streaming_chat_message_content import StreamingChatMessageContent
 from semantic_kernel.core_plugins import MathPlugin, TimePlugin
 from semantic_kernel.functions import KernelArguments
-from semantic_kernel.utils.settings import openai_settings_from_dot_env
 
 if TYPE_CHECKING:
     from semantic_kernel.functions import KernelFunction
@@ -38,12 +37,10 @@
 kernel = Kernel()
 
 # Note: the underlying gpt-35/gpt-4 model version needs to be at least version 0613 to support tools.
-api_key, org_id = openai_settings_from_dot_env()
 kernel.add_service(
     OpenAIChatCompletion(
         service_id="chat",
         ai_model_id="gpt-3.5-turbo-1106",
-        api_key=api_key,
     ),
 )
 

python/samples/concepts/auto_function_calling/chat_gpt_api_function_calling.py

@@ -7,7 +7,6 @@
 from semantic_kernel.connectors.ai.function_call_behavior import FunctionCallBehavior
 from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
 from semantic_kernel.contents import ChatHistory
-from semantic_kernel.utils.settings import azure_openai_settings_from_dot_env_as_dict
 
 logging.basicConfig(level=logging.WARNING)
 
@@ -24,7 +23,7 @@
 
 service_id = "chat-gpt"
 chat_service = AzureChatCompletion(
-    service_id=service_id, **azure_openai_settings_from_dot_env_as_dict(include_api_version=True)
+    service_id=service_id,
 )
 kernel.add_service(chat_service)
 

python/samples/concepts/chat_completion/azure_chat_gpt_api.py

@@ -6,7 +6,6 @@
 from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
 from semantic_kernel.contents import ChatHistory
 from semantic_kernel.prompt_template import InputVariable, PromptTemplateConfig
-from semantic_kernel.utils.settings import openai_settings_from_dot_env
 
 prompt = """
 ChatBot can have a conversation with you about any topic.
@@ -21,11 +20,8 @@
 
 kernel = Kernel()
 
-api_key, org_id = openai_settings_from_dot_env()
 service_id = "chat"
-kernel.add_service(
-    OpenAIChatCompletion(service_id=service_id, ai_model_id="gpt-3.5-turbo-1106", api_key=api_key, org_id=org_id)
-)
+kernel.add_service(OpenAIChatCompletion(service_id=service_id))
 
 settings = kernel.get_prompt_execution_settings_from_service_id(service_id)
 settings.max_tokens = 2000

python/samples/concepts/chat_completion/chat.py

@@ -6,7 +6,6 @@
 from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
 from semantic_kernel.contents import ChatHistory
 from semantic_kernel.functions import KernelArguments
-from semantic_kernel.utils.settings import openai_settings_from_dot_env
 
 system_message = """
 You are a chat bot. Your name is Mosscap and
@@ -19,11 +18,8 @@
 
 kernel = Kernel()
 
-api_key, org_id = openai_settings_from_dot_env()
 service_id = "chat-gpt"
-kernel.add_service(
-    OpenAIChatCompletion(service_id=service_id, ai_model_id="gpt-3.5-turbo", api_key=api_key, org_id=org_id)
-)
+kernel.add_service(OpenAIChatCompletion(service_id=service_id))
 
 settings = kernel.get_prompt_execution_settings_from_service_id(service_id)
 settings.max_tokens = 2000