Python: Add the Python process framework

[moonbox3]

Motivation and Context

An initial PR to add the foundational pieces of the Python Process framework, which holds it design to be similar to dotnet in that step types are added to a process builder, and later on, when the step is run, it is first instantiated and the proper state is provided.

Description

Adding the initial process framework components:

• Closes Python: support Process framework #9354

TODO

• more unit tests will be added to increase the code coverage. Currently there are several files with no (or low) code coverage.
• more samples will either be added to this PR or a subsequent PR

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[markwallace-microsoft]

Python Test Coverage Report

File	Stmts	Miss	Cover	Missing
semantic_kernel
kernel.py	199	47	76%	148, 159, 163, 313–316, 423, 437–480
semantic_kernel/agents/group_chat
agent_chat.py	124	2	98%	78, 171
agent_group_chat.py	100	2	98%	151, 201
broadcast_queue.py	72	1	99%	35
semantic_kernel/agents/open_ai
assistant_content_generation.py	141	9	94%	97–98, 329–337, 379, 381
open_ai_assistant_base.py	467	8	98%	260, 338–339, 747, 868, 871, 945, 1007
semantic_kernel/connectors/ai
chat_completion_client_base.py	116	2	98%	382, 392
completion_usage.py	8	1	88%	17
semantic_kernel/connectors/ai/anthropic/services
anthropic_chat_completion.py	176	5	97%	147, 165, 169, 223, 419
semantic_kernel/connectors/ai/azure_ai_inference/services
azure_ai_inference_chat_completion.py	119	7	94%	120, 146–149, 159, 180, 202
azure_ai_inference_text_embedding.py	41	1	98%	87
semantic_kernel/connectors/ai/bedrock/services
bedrock_chat_completion.py	136	14	90%	117, 138, 163, 167–170, 228, 246–265, 324
bedrock_text_completion.py	57	2	96%	95, 118
bedrock_text_embedding.py	45	1	98%	94
semantic_kernel/connectors/ai/bedrock/services/model_provider
bedrock_ai21_labs.py	13	1	92%	67
bedrock_anthropic_claude.py	12	1	92%	54
bedrock_cohere.py	20	1	95%	75
utils.py	80	20	75%	68, 71, 102, 106–115, 132–150, 171–174
semantic_kernel/connectors/ai/embeddings
embedding_generator_base.py	8	1	88%	50
semantic_kernel/connectors/ai/google/google_ai/services
google_ai_chat_completion.py	119	4	97%	127, 153, 176, 178
google_ai_text_completion.py	63	2	97%	98, 121
utils.py	65	3	95%	139, 159–164
semantic_kernel/connectors/ai/google/vertex_ai/services
utils.py	66	3	95%	140, 160–165
vertex_ai_chat_completion.py	119	4	97%	121, 147, 170, 172
vertex_ai_text_completion.py	62	2	97%	95, 116
semantic_kernel/connectors/ai/hugging_face/services
hf_text_completion.py	60	3	95%	103, 112, 127
hf_text_embedding.py	32	5	84%	79–83
semantic_kernel/connectors/ai/mistral_ai/services
mistral_ai_chat_completion.py	118	7	94%	118–121, 307–310
semantic_kernel/connectors/ai/ollama/services
ollama_chat_completion.py	60	7	88%	72–73, 95–98, 108, 143
ollama_text_completion.py	57	7	88%	68–69, 90–93, 103, 131
ollama_text_embedding.py	41	2	95%	65–66
semantic_kernel/connectors/ai/onnx
utils.py	53	3	94%	50–51, 112
semantic_kernel/connectors/ai/onnx/services
onnx_gen_ai_chat_completion.py	72	7	90%	67–68, 98, 122, 167, 173, 179
onnx_gen_ai_completion_base.py	58	21	64%	59–71, 79–90
onnx_gen_ai_text_completion.py	46	5	89%	54–55, 87, 117, 133
semantic_kernel/connectors/ai/open_ai/prompt_execution_settings
open_ai_prompt_execution_settings.py	94	1	99%	112
semantic_kernel/connectors/ai/open_ai/services
azure_chat_completion.py	107	5	95%	118, 123, 157, 166, 169
azure_text_completion.py	28	2	93%	82, 87
azure_text_embedding.py	30	2	93%	84, 89
open_ai_chat_completion_base.py	127	5	96%	71, 121, 141, 177, 287
open_ai_handler.py	63	3	95%	86, 95–96
open_ai_text_completion_base.py	80	2	98%	56, 161
semantic_kernel/connectors/ai/open_ai/settings
azure_open_ai_settings.py	22	1	95%	99
semantic_kernel/connectors/memory/azure_ai_search
azure_ai_search_collection.py	87	2	98%	150, 152
semantic_kernel/connectors/memory/redis
redis_collection.py	160	2	99%	146, 316
utils.py	45	11	76%	145–146, 164, 166, 173–188
semantic_kernel/connectors/memory/weaviate
utils.py	60	4	93%	78, 82, 110, 249
weaviate_collection.py	115	25	78%	144–153, 157–177, 181–186
weaviate_store.py	44	9	80%	106–114, 118–123
semantic_kernel/connectors/openapi_plugin
openapi_manager.py	58	2	97%	110–111
openapi_parser.py	88	2	98%	71, 128
openapi_runner.py	105	2	98%	181–182
semantic_kernel/connectors/openapi_plugin/models
rest_api_operation.py	129	1	99%	242
semantic_kernel/contents
function_call_content.py	100	2	98%	185, 213
streaming_chat_message_content.py	68	1	99%	210
streaming_content_mixin.py	39	2	95%	37, 64
semantic_kernel/core_plugins/sessions_python_tool
sessions_python_plugin.py	134	8	94%	69, 82–91, 99
sessions_python_settings.py	39	4	90%	84–87
semantic_kernel/data
vector_store_record_collection.py	249	19	92%	408, 468–472, 480–484, 524–527, 534–537
vector_store_record_utils.py	26	2	92%	50, 52
semantic_kernel/functions
kernel_function_decorator.py	98	1	99%	102
kernel_function_from_method.py	96	1	99%	153
kernel_function_from_prompt.py	154	7	95%	165–166, 180, 201, 219, 239, 322
kernel_function_log_messages.py	36	6	83%	37–43
kernel_plugin.py	187	2	99%	472, 475
semantic_kernel/planners
plan.py	234	45	81%	54, 163–165, 197, 214–227, 264, 269, 277–278, 288–291, 308, 313, 329, 332–337, 355, 360, 363, 365, 372, 386–388, 393–397
semantic_kernel/planners/function_calling_stepwise_planner
function_calling_stepwise_planner.py	116	4	97%	145, 189–190, 198
semantic_kernel/planners/sequential_planner
sequential_planner.py	64	6	91%	71, 75, 109, 125, 134–135
sequential_planner_extensions.py	50	9	82%	31–32, 56, 110–124
sequential_planner_parser.py	77	12	84%	66–74, 93, 117–120
semantic_kernel/processes
process_builder.py	68	39	43%	43–52, 56–58, 64–74, 78–85, 89–92, 96–100, 105, 109–114
process_end_step.py	19	2	89%	37, 41
process_function_target_builder.py	25	3	88%	37–40
process_step_builder.py	105	24	77%	44, 89, 103, 110–123, 135–142, 151, 160–169, 178, 192, 209
process_step_edge_builder.py	34	3	91%	31, 46, 56
process_types.py	25	1	96%	35
semantic_kernel/processes/local_runtime
local_kernel_process.py	20	2	90%	23, 30
local_kernel_process_context.py	32	2	94%	66–67
local_process.py	134	52	61%	92, 102, 120–130, 163–190, 194–199, 203, 207–213, 217–227, 231–232
local_step.py	178	114	36%	61, 72, 81–169, 173, 177, 181–182, 187–249, 253–270, 274–277, 281–284, 288–297, 303–306, 310–312
semantic_kernel/schema
kernel_json_schema_builder.py	129	9	93%	53, 90, 186, 194, 205, 213, 228, 232–233
semantic_kernel/services
ai_service_client_base.py	22	1	95%	64
semantic_kernel/template_engine/blocks
code_block.py	77	1	99%	119
named_arg_block.py	43	1	98%	98
semantic_kernel/utils/authentication
entra_id_authentication.py	15	2	87%	26, 38
semantic_kernel/utils/telemetry
user_agent.py	16	2	88%	18–19
semantic_kernel/utils/telemetry/model_diagnostics
decorators.py	171	4	98%	372–375
TOTAL	13305	685	95%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
2760	4 💤	0 ❌	0 🔥	1m 8s ⏱️

[eavanvalkenburg]

first run through, looks pretty good but I would love if we can make things less verbose and easier to dev with!

[rog555]

@moonbox3 this is super exciting, any thoughts on the implementation being abstracted enough to be able to use a non Dapr/Orleans workflow backend? For example Azure Pub/Sub + Functions or something equivilent (eg this sort of thing)

[moonbox3]

@moonbox3 this is super exciting, any thoughts on the implementation being abstracted enough to be able to use a non Dapr/Orleans workflow backend? For example Azure Pub/Sub + Functions or something equivilent (eg this sort of thing)

Hi @rog555, we're already at a level of abstraction where it's pretty easy to BOYB (bring your own backend). This initial runtime, as you can see, is the local runtime. Dapr will come next. Once this framework is in main, we would love for you to experiment adding your own runtime/backend.

[rog555]

Thanks, roughly when do you think 1.12.0 will be available in the wild - before or after ignite? 😃👍

[moonbox3]

Thanks, roughly when do you think 1.12.0 will be available in the wild - before or after ignite? 😃👍

Hi @rog555, definitely before! Trying to get 1.12.0 out this week, if possible. I need to add a few more samples and do some validations like running a process as a subprocess, but should be able to get that in by tomorrow/Thursday.