Add comprehensive prompt customization documentation

[devin-ai-integration]

Add Comprehensive Prompt Customization Documentation

Overview

This PR addresses issue #3045 by providing comprehensive documentation explaining how CrewAI generates system and user prompts and how users can customize them for precise control over agent behavior.

Changes Made

Documentation Added

• New Documentation File: docs/how-to/customize-prompts.mdx
  • Explains CrewAI's template-based prompt generation system
  • Documents the core prompt components stored in src/crewai/translations/en.json
  • Shows how prompts are assembled using the Prompts class
  • Covers both regular Agent and LiteAgent prompt generation
  • Provides detailed customization examples

Key Topics Covered

1. Understanding Prompt Generation

   • Core prompt components (role_playing, tools, no_tools, task, format instructions)
   • Prompt assembly process for different agent types
   • System/user prompt split functionality

2. Basic Prompt Customization

   • Custom system and prompt templates using {{ .System }}, {{ .Prompt }}, {{ .Response }} placeholders
   • Template parameter usage (system_template, prompt_template, response_template)
   • System/user prompt separation with use_system_prompt=True

3. Output Format Customization

   • Structured output with Pydantic models
   • Custom format instructions using output_format parameter
   • Response format control for LiteAgents

4. Stop Words Configuration

   • Default stop words behavior ("\nObservation:" for tool-enabled agents)
   • Custom stop words via response templates
   • Integration with litellm completion calls

5. Advanced Examples

   • Multi-language support
   • Domain-specific formatting (medical reports example)
   • Complex workflow integration with Flows
   • Best practices for production environments

Tests Added

• Comprehensive Test Suite: tests/test_prompt_customization_docs.py
  • Validates all documentation examples work correctly
  • Tests custom template functionality
  • Verifies system/user prompt split behavior
  • Tests structured output with Pydantic models
  • Validates stop words configuration
  • Tests LiteAgent prompt customization
  • Covers advanced scenarios like multi-language support

Answers to Issue #3045 Questions

The documentation directly addresses all questions from the original issue:

1. "How the system and user prompts are generated?"

   • Documented the template system in src/crewai/translations/en.json
   • Explained prompt assembly in src/crewai/utilities/prompts.py
   • Covered LiteAgent generation in src/crewai/lite_agent.py

2. "How can we modify the default text in system and user prompt?"

   • Showed custom template usage with system_template, prompt_template, response_template
   • Demonstrated template placeholder system
   • Provided practical examples

3. "What is the argument to pass to get format in the system prompt?"

   • Documented output_format and output_pydantic parameters
   • Explained formatted_task_instructions slice usage
   • Showed structured output examples

4. "How to modify litellm completion stop arguments?"

   • Explained stop words configuration in src/crewai/agent.py
   • Showed how response_template affects stop words
   • Documented the observation slice control

Testing

• ✅ Created comprehensive test suite with 15+ test cases
• ✅ Validated all documentation examples work correctly
• ✅ Tested basic functionality with temporary validation script
• ✅ All examples are practical and copy-pasteable

Technical Implementation

The documentation follows CrewAI's established patterns:

• Uses MDX format consistent with existing docs
• Emphasizes precision, accuracy, and fine-grained control
• Focuses on technical substance for engineers
• Includes security considerations and best practices
• Provides real-world application examples

Link to Devin Run

https://app.devin.ai/sessions/cd56954610384d9bb432a2c67e1801e2

Requested by

João (joao@crewai.com)

This comprehensive documentation gives users precise control over agent behavior while maintaining the reliability and consistency that CrewAI is known for in production environments.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment for PR #3046: Add Comprehensive Prompt Customization Documentation

Overview

This pull request significantly enhances CrewAI's documentation by providing comprehensive guidance and test validation for the prompt customization system. The changes include two new files:

• docs/how-to/customize-prompts.mdx
• tests/test_prompt_customization_docs.py

Strengths

Documentation (customize-prompts.mdx)

1. Structural Clarity: The document is well-organized, providing clear headings that guide the user through different aspects of prompt customization.
2. Coverage: It effectively covers various customization features, which is crucial for user understanding and implementation.
3. Code Examples: The usage of code snippets facilitates learning and application, making the documentation user-friendly.

Tests (test_prompt_customization_docs.py)

1. Comprehensive Test Coverage: The file ensures that all documentation examples are effectively validated against expected outcomes, establishing a robust testing foundation.
2. Effective Use of Pytest: The utilization of pytest fixtures and structured classes enhances code organization and testing efficiency.

Suggested Improvements

Documentation Enhancements

1. Markdown Formatting: Add categoryId and priority metadata for improved organization.

   ---
   title: "Customize Agent Prompts"
   description: "Learn how to customize system and user prompts..."
   categoryId: "how-to-guides"
   priority: 1
   ---

2. Code Block Language Specification: Include showLineNumbers for clearer code examples.

   ```python showLineNumbers
   agent = Agent(...)

3. Cross-References: Use markdown links for better internal navigation.

   See the [Troubleshooting](#troubleshooting) section for common issues.

Test File Enhancements

1. Documentation of Test Methods: Elaborate on test purposes for clarity on what is being validated.

   def test_custom_system_and_prompt_templates(self):
       """Test basic custom template functionality.
       Validates:
       - Custom system template assignment
       - Custom prompt template assignment
       - Custom response template assignment
       - System prompt flag configuration
       """

2. Organize Test Data: Consider moving test data/models to the top or separating them into a distinct fixtures file for better readability.

   # Move the ResearchOutput class to the top of the file or into a separate fixtures file

3. More Specific Mock Assertions: Enhance mock assertions for greater clarity and reliability.

   mock_create.assert_called_once_with(mock_task)
   assert mock_create.call_args[0][0] == mock_task

4. Test Independence: Implement fixtures for test case isolation.

   @pytest.fixture(autouse=True)
   def setup(self):
       self.i18n = I18N()

Additional Considerations

• Security: Add guidance on template injection prevention and input validation to ensure user safety.
• Edge Cases: Encourage the addition of tests for edge cases and performance benchmarks to comprehensively validate functionality and efficiency.

Conclusion

This PR presents an excellent addition to CrewAI’s documentation and testing framework for prompt customization. By addressing the suggested improvements, we can enhance its usability, security, and maintainability further. Overall, the changes are approved with these recommendations for refinement.

This type of structured feedback can guide iterative improvements while ensuring that the documentation remains aligned with user needs and best practices in coding standards.

[mplachta]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment for PR #3046: Add Comprehensive Prompt Customization Documentation

Thank you for this substantial and well-crafted contribution. This PR significantly enhances CrewAI by providing an exhaustive, clear, and practical prompt customization guide paired with extensive tests verifying the documented examples. Below is a detailed review to summarize key findings, plus specific improvement suggestions to further strengthen the deliverable.

---

Summary of Key Findings

• Comprehensive Documentation:
  The new docs/how-to/customize-prompts.mdx is an impressively detailed resource that thoroughly explains CrewAI’s prompt generation architecture, including template structures, prompt assembly for different agent types, output formatting, and stop word customization. The inclusion of real-world, practical code examples and advanced scenarios (multi-language, domain-specific, complex flows) greatly enhances developer usability.

• Well-Structured and Clear Content:
  Documentation sections are logically organized, following a natural progression from explanation to examples, to best practices and troubleshooting. Coverage spans basic usage to advanced customization, helping both new and experienced users.

• Rigorous Test Suite:
  The tests/test_prompt_customization_docs.py suite validates that the documentation examples function as described, providing executable verification. It spans template overrides, system/user prompts, Pydantic structured outputs, response formatting, stop words, LiteAgent prompts, internationalization, and prompt assembly logic. This strengthens confidence in correctness and future maintainability.

• Security Awareness:
  Security considerations are briefly mentioned, particularly around validating inputs and mitigating prompt injection risks, showing prudent design thinking.

• Internationalization Support:
  The focus on template slices sourced from translations/en.json and tests verifying language customization highlights careful i18n integration.

---

Specific Improvement Suggestions

1. Enhance Markdown Code Block Formatting

Several templates embedded within triple quotes ("""...""") contain curly braces and placeholders that can appear unclear or risk poor Markdown rendering.

Suggestion: Use fenced code blocks with explicit language identifiers for improved syntax highlighting and clarity in docs. For example:

Additional context: You are working in a production environment.
Always prioritize accuracy and provide detailed explanations.

Or for Python samples:

agent = Agent(
    role="Data Analyst",
    goal="Analyze data with precision",
    backstory="Experienced analyst.",
    system_template=system_template,
    prompt_template=prompt_template,
    response_template=response_template,
    use_system_prompt=True
)

This will improve readability and user experience on rendered documentation sites.

---

2. Add Explicit Warnings About Stop Sequence Pitfalls

In the Stop Words Configuration section, it is important to warn users about the risks of premature truncation when an ill-chosen stop word sequence appears within valid output text.

Proposed addition:

⚠️ Warning: If your stop sequence (e.g., ---END---) can appear naturally within the model’s response, this may cause premature output truncation. Always select distinctive, unlikely-to-occur sequences for stopping generation.

---

3. Expand Security Best Practices Regarding Template Inputs

The security note is a good start but could be made more explicit about the dangers of injecting unsanitized user input into template placeholders.

Proposed addition:

🛡️ Security Tip: Avoid injecting raw or untrusted inputs directly into prompt templates without validation or sanitization. Use controlled data or escape sequences appropriately to prevent prompt injection attacks or unintended model behavior.

---

4. Consistently Document Default Parameter Behaviors

When describing parameters like system_template, prompt_template, and response_template, please explicitly state what defaults are used if these parameters are omitted (typically the defaults from translations/en.json).

Example:

If system_template is not provided, the default template from translations/en.json is used.

This clarity helps users understand the fallback behavior.

---

5. Improve Header Hierarchy Consistency and Navigation

Some header levels jump unevenly (e.g., from ### to #### without intermediate levels). Consider refining header granularity for better table of contents navigation and readability, especially in sections with nested topics.

---

6. Cross-Link To Relevant API Reference Documentation

For better maintainability and developer guidance, link mentions of Agent, LiteAgent, and other CrewAI classes to their respective API reference documents.

Example:
See the Agent API Reference for complete details on constructor parameters and options.

---

7. Expand Troubleshooting Section with Additional Guidance

Add actionable steps, besides existing notes, such as:

• Reverting to default templates to isolate customization errors
• Verifying prompt payloads in verbose/debug mode
• Testing stop word effects carefully on output truncation

---

8. Suggestions for Test Suite Enhancements

• Add Negative Test Cases: Include tests for default parameter usage (no custom templates) and for incorrect or incomplete template definitions to ensure graceful fallback or errors.
• Refine Test Naming: Consider BDD-style names like test_stop_words_applied_when_response_template_set for clearer intent.
• Parametrize Some Tests: For repeated patterns (e.g., agent role/goal/backstory), use pytest.mark.parametrize to reduce redundancy.
• Use Context Managers for Patching: To avoid side effects and ensure safe restoration even on failures, use with patch(...) as where patching is done.

Example parametrized test snippet:

import pytest

@pytest.mark.parametrize("role,goal,backstory", [
    ("Analyst", "Analyze data", "Expert analyst"),
    ("Researcher", "Find facts", "Experienced researcher"),
])
def test_agent_initialization(role, goal, backstory):
    agent = Agent(role=role, goal=goal, backstory=backstory)
    assert agent.role == role
    assert agent.goal == goal
    assert agent.backstory == backstory

---

Historical Context and Related Learnings

While this PR is new, its design and testing patterns reflect maturity likely gained from prior CrewAI PRs refining the prompt customization system, prompt assembly logic, and structured output handling. The solid test coverage and tight coupling between documentation and tested examples are best practices that improve long-term project health and feature extensibility.

The approach taken here would benefit future enhancements such as:

• More granular template fragments for dynamic prompt assembly
• Improved security guards around template injection
• Enhanced tooling/support for multi-lingual prompt templates
• Expanded advanced workflow and integration samples
• Continuous integration checks validating prompt correctness

---

Implications for Related Files and Future Work

• This documentation and test suite path the way for developers modifying prompt logic or adding new agent types to understand expected behaviors and customization avenues thoroughly.
• The extensive testing will help ensure that future prompt system refactors or feature additions do not inadvertently break documented behaviors or examples.
• The stop word customization mechanism and structured output format handling should be scrutinized if modified going forward to avoid regressions or unexpected semantic changes.
• Collaboration between doc writers and developers should continue to keep documentation living and matching codebase evolution.

---

Conclusion

This PR delivers a high-value, meticulously detailed documentation and validation effort for CrewAI’s prompt customization feature. The code and docs align well with best engineering and documentation practices. The suggestions above are mainly to further polish clarity, usability, and robustness but do not reveal any critical flaws.

I recommend moving forward with merging after optionally applying the noted improvements, particularly on markdown code block formatting, explicit warnings, and enriched security tips.

Thank you for this excellent, well-tested contribution that will significantly aid CrewAI users and maintainers!

---

Please reach out if you want detailed inline comments or help implementing recommended refinements.