Support list of YAML files for agents_config and tasks_config in CrewBase

[devin-ai-integration]

Support list of YAML files for agents_config and tasks_config in CrewBase

Description

This PR implements the feature requested in issue #2721. It allows the agents_config and tasks_config attributes in the CrewBase decorator to accept a list of YAML file paths instead of just a single path string.

Changes

• Modified the CrewBase decorator to handle both string paths and lists of paths
• Added a new helper method to load and merge configurations from multiple YAML files
• Updated the __init__ method to use the new helper method
• Added tests to verify the new functionality

How to use

Previously:

@CrewBase
class MyCrew:
    agents_config = "config/agents.yaml"
    tasks_config = "config/tasks.yaml"

Now you can also do:

@CrewBase
class MyCrew:
    agents_config = ["config/agents1.yaml", "config/agents2.yaml"]
    tasks_config = ["config/tasks1.yaml", "config/tasks2.yaml"]

Configurations from multiple files are merged, with later files overriding earlier ones for duplicate keys.

Resolves #2721

Link to Devin run: https://app.devin.ai/sessions/e13afb9d30dd405a8b0066212f335934
Requested by: Joe Moura (joao@crewai.com)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[mplachta]

Disclaimer: This review was made by a crew of AI Agents.

---

Code Review for PR #2722 — Multi-YAML Config Support in CrewBase

Summary

This PR introduces the capability to specify multiple YAML configuration files for agents_config and tasks_config class attributes in CrewBase. It enhances flexibility by allowing users to layer configurations, where subsequent YAML files override previous ones’ settings at the top dictionary level. The major change is the addition of a helper function, load_and_merge_yaml_configs, which merges multiple YAML dictionaries with simple shallow merging. The patch also adds realistic multi-file YAML fixtures and a comprehensive test validating merge behavior and correct instantiation of agents and tasks from the combined configurations.

The implementation is robust, idiomatic, and well-covered with integration tests. The new feature will allow complex layered project configurations with explicit override support, a valuable enhancement for CrewBase users.

---

Detailed Findings and Suggestions

1. Code Quality and Improvements

a) Normalization of Input Types for Config Paths

Current:
agents_config_paths and tasks_config_paths are conditionally assigned either a list of Path or a single Path depending on whether original_*_config_path is a list or a single element. This results in inconsistent types passed to the merge method.

Issue:
Such inconsistent types can cause subtle bugs and require extra type checking downstream.

Recommendation:
Normalize always to a list of Paths, even when a single config path is supplied:

if isinstance(self.original_agents_config_path, list):
    agents_config_paths = [self.base_directory / p for p in self.original_agents_config_path]
else:
    agents_config_paths = [self.base_directory / self.original_agents_config_path]

# Same for tasks

This normalization simplifies load_and_merge_yaml_configs to expect only List[Path], improving predictability and type safety.

---

b) Merge Behavior: Shallow vs Deep Merge

Current:
The merge approach performs a shallow dictionary update:

• At the top level, keys are merged.
• If a key exists in both dicts and their values are dicts, only one-level .update() is done on the nested dict.

Implications:
If nested dicts are deeper than one level, deeper keys are replaced rather than merged, potentially unintentionally overwriting nested config sections.

Options:

• (Preferred) Document the shallow merge behavior clearly in the method’s docstring to set user expectations.
• (Optional Future) Implement a recursive deep merge helper if use cases require nested dict merging, e.g.:

def deep_merge(dict1, dict2):
    result = dict1.copy()
    for k, v in dict2.items():
        if k in result and isinstance(result[k], dict) and isinstance(v, dict):
            result[k] = deep_merge(result[k], v)
        else:
            result[k] = v
    return result

This can replace the current .update() call to avoid overwriting nested keys.

---

c) Error Handling and Logging

Current:
In load_yaml, a FileNotFoundError triggers print(f"File not found: {config_path}") followed by raise. This is minimal and noisy in library usage.

Recommendation:
Adopt logging for errors instead of print(). For example:

import logging

def load_yaml(self, config_path: Path):
    try:
        # Load YAML
    except FileNotFoundError:
        logging.error(f"Configuration YAML file not found: {config_path}")
        raise

This allows better control of error visibility and integrates with existing logging systems.

---

d) Type Annotations and Python Version Compatibility

• The current annotation list[Path] | Path requires Python 3.10+.
• If the project supports earlier versions, consider using:

from typing import Union, List

def load_and_merge_yaml_configs(self, config_paths: Union[List[Path], Path]) -> dict:
    ...

• Ensure consistent typing with return types (-> dict) and throughout helper functions. This improves readability and IDE support.

---

2. Tests and YAML Configurations

• The test test_multiple_yaml_configs in tests/project_test.py is well-designed, covering:

  • Multiple YAML loading
  • Key overriding behavior
  • Agent and task instantiation using merged configs
  • Mutation safety using .copy()

• The YAML fixtures under tests/config/multi/ accurately simulate layered config scenarios and key conflict cases.

Suggestions:

• Consider adding parameterized tests for variations such as:
  • Single config file only (fallback behavior)
  • More complex nested dictionary configs, to preemptively test deep merge scenarios if supported in the future.
• Review the necessity of the @pytest.mark.vcr marker for this test to avoid unnecessary external dependencies unless required.

---

3. Documentation and Usage Notes

• The new feature should be explicitly documented in user-facing documentation and code docstrings, especially:
  • That users may supply either a single YAML path or a list of paths.
  • Merging semantics: later files override earlier ones at the top level, with shallow merging of nested dicts.
  • Guidance about nested dict behavior and potential limitations.

---

4. Historical Context and Learned Patterns

• This PR continues a trend in this repo of extending configuration flexibility, providing layered configs and override mechanisms.
• Previous single-config loading efforts laid foundation that this multi-file support builds on gracefully.
• The use of shallow dictionary merging is a common practical choice balancing simplicity and expected usage.
• The addition of comprehensive integration tests aligns with patterns of robust release quality seen in recent PRs.

---

5. Summary Table of Suggestions

Aspect	Current Behavior	Suggestion / Improvement
Config Path Type Handling	Mixed list or single Path	Normalize to always list[Path]
Merge Depth	Shallow dict merge	Document clearly or optionally implement deep merge
Error Handling	print + raise on FileNotFoundError	Use logging with meaningful error messages
Type Annotations	Python 3.10+ union syntax	Verify Python version support or fallback to Union + List
Testing	Good coverage with base and override scenarios	Add nested dict test cases and parametrize input
Documentation	Docstrings present but no user doc update noted	Add user-facing docs explaining multi-config usage and merge rules

---

Code Snippet Example: Improved Normalization and Deep Merge

def normalize_to_path_list(base_directory, paths):
    if isinstance(paths, (list, tuple)):
        return [base_directory / p for p in paths]
    else:
        return [base_directory / paths]

def deep_merge(dict1: dict, dict2: dict) -> dict:
    result = dict1.copy()
    for key, value in dict2.items():
        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
            result[key] = deep_merge(result[key], value)
        else:
            result[key] = value
    return result

def load_and_merge_yaml_configs(self, config_paths: list[Path]) -> dict:
    result = {}
    for path in config_paths:
        config = self.load_yaml(path)
        if config:
            for key, val in config.items():
                if key in result and isinstance(result[key], dict) and isinstance(val, dict):
                    result[key] = deep_merge(result[key], val)
                else:
                    result[key] = val
    return result

---

Final Conclusion

This PR delivers a meaningful and user-oriented feature with a clean implementation and excellent tests. The improvements suggested here primarily target maintainability, robustness, logging practices, and clarity about merge semantics. By normalizing types and enhancing error handling, the feature will be even more solid and friendly for future developers. The addition of user documentation will help adoption.

Overall, this is a valuable and well-crafted contribution to the crewAIInc/crewAI codebase.

---