Update haiway and evals concurrency

[KaQuMiQ]

No description provided.

[coderabbitai]

Walkthrough

This change updates the default UV_VERSION in the Makefile from 0.7.6 to 0.7.19 and increments the project version in pyproject.toml from 0.75.4 to 0.75.5. The dependency on the "haiway" package is updated from 0.24.0 to 0.24.2, and a new optional dependency group "openai_realtime" is added. In the codebase, two new symbols, execute_concurrently and stream_concurrently, are re-exported from the haiway module in draive/__init__.py. The evaluation logic in scenario.py and suite.py is refactored to use execute_concurrently for controlled concurrency, and both now accept a concurrent_tasks parameter. Additionally, EvaluationSuite._evaluate now supports random sampling of cases by integer count or fraction.

Possibly related PRs

• Refine context attributes and model interfaces #313: Updates the UV_VERSION variable in the Makefile, similar to this PR, but targets a different version, indicating ongoing version management for the uv tool.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ca387e3 and 4185750.

⛔ Files ignored due to path filters (1)

• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (5)

• Makefile (1 hunks)
• pyproject.toml (3 hunks)
• src/draive/__init__.py (4 hunks)
• src/draive/evaluation/scenario.py (2 hunks)
• src/draive/evaluation/suite.py (4 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

`**/__init__.py`: Put exported symbols into __init__.py

**/__init__.py: Put exported symbols into init.py

📄 Source: CodeRabbit Inference Engine (CLAUDE.md)

List of files the instruction was applied to:

• src/draive/__init__.py

`**/*.py`: Follow Ruff import ordering (standard library, third party, local) Us...

**/*.py: Follow Ruff import ordering (standard library, third party, local)
Use Python 3.12+ type features (type unions with |, generic syntax)
Use base and abstract types like Sequence or Iterable instead of concrete types
Use custom exceptions for specific errors
Format code with Ruff
Run linters (Ruff + Bandit + Pyright strict mode)

📄 Source: CodeRabbit Inference Engine (CLAUDE.md)

List of files the instruction was applied to:

• src/draive/__init__.py
• src/draive/evaluation/scenario.py
• src/draive/evaluation/suite.py

🧠 Learnings (6)

📓 Common learnings

Learnt from: KaQuMiQ
PR: miquido/draive#338
File: src/draive/lmm/__init__.py:1-2
Timestamp: 2025-06-16T10:28:07.434Z
Learning: The draive project requires Python 3.12+ as specified in pyproject.toml with "requires-python = ">=3.12"" and uses Python 3.12+ specific features like PEP 695 type aliases and generic syntax extensively throughout the codebase.

pyproject.toml (1)

Learnt from: KaQuMiQ
PR: miquido/draive#338
File: src/draive/lmm/__init__.py:1-2
Timestamp: 2025-06-16T10:28:07.434Z
Learning: The draive project requires Python 3.12+ as specified in pyproject.toml with "requires-python = ">=3.12"" and uses Python 3.12+ specific features like PEP 695 type aliases and generic syntax extensively throughout the codebase.

src/draive/__init__.py (2)

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: draive builds on top of haiway and exports its symbols

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: Applies to draive/**/*.py : Use absolute imports from draive package

Makefile (1)

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: Sync dependencies with uv lock file

src/draive/evaluation/scenario.py (2)

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: Applies to draive/**/*.py : Use async context managers for resource management and cleanup

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: Applies to draive/**/*.py : Use Protocol and runtime_checkable for function protocols

src/draive/evaluation/suite.py (3)

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: Applies to tests/**/*.py : Tests are in tests/ directory and use pytest with async support

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: Applies to draive/**/*.py : Use Protocol and runtime_checkable for function protocols

Learnt from: CR
PR: miquido/draive#0
File: CLAUDE.md:0-0
Timestamp: 2025-07-01T09:39:43.008Z
Learning: Applies to draive/**/*.py : Use async context managers for resource management and cleanup

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: test (macos-latest, 3.13)
• GitHub Check: test (ubuntu-latest, 3.13)

🔇 Additional comments (14)

Makefile (1)

13-13: UV version bump validated

• Makefile (line 13): UV_VERSION := 0.7.19 is correct—0.7.19 is the latest stable release on PyPI as of July 2025.
  No further action required.

pyproject.toml (3)

8-8: LGTM! Version bump aligns with the changes.

The version increment from 0.75.4 to 0.75.5 is appropriate for the feature additions and dependency updates in this PR.

---

24-24: LGTM! Haiway dependency update supports new concurrency features.

The update from haiway 0.24.0 to 0.24.2 provides the new concurrency utilities (execute_concurrently and stream_concurrently) that are being used in the evaluation modules.

---

35-35: LGTM! Well-structured optional dependency group.

The new openai_realtime dependency group properly includes the OpenAI package with realtime support and tiktoken with appropriate version constraints.

src/draive/__init__.py (2)

36-36: LGTM! Proper import of new concurrency utilities.

The imports of execute_concurrently and stream_concurrently from haiway follow the existing alphabetical ordering pattern and align with the project's practice of exposing haiway symbols for easier usage.

Also applies to: 49-49

---

396-396: LGTM! Correct addition to public API.

Both concurrency utilities are properly added to the __all__ tuple in alphabetical order, making them available as part of the draive package's public API.

Also applies to: 416-416

src/draive/evaluation/scenario.py (3)

4-4: LGTM! Proper import addition for concurrency control.

The import of execute_concurrently aligns with the new concurrency functionality being introduced to the evaluation scenario.

---

86-86: LGTM! Sensible default for concurrent task limit.

The concurrent_tasks: int = 2 parameter provides good control over concurrency with a conservative default that balances performance and resource usage.

---

89-100: LGTM! Well-implemented concurrency control.

The implementation properly:

• Defines an inner execute function to wrap evaluator calls
• Uses execute_concurrently with the specified task limit
• Maintains the same return type and behavior as the previous implementation

This provides better resource management compared to uncontrolled asyncio.gather.

src/draive/evaluation/suite.py (5)

1-1: LGTM! Random module import supports new sampling functionality.

The import is needed for the new random sampling feature for evaluation cases.

---

8-8: LGTM! Consistent import pattern for concurrency control.

The import of execute_concurrently aligns with the concurrency improvements being made across evaluation modules.

---

183-196: LGTM! Consistent concurrency implementation.

The concurrency control implementation matches the pattern used in scenario.py, providing consistent behavior across evaluation modules with proper task limiting.

---

310-312: LGTM! Expanded type annotation supports new sampling modes.

The type annotation correctly includes int and float to support the new random sampling functionality while maintaining backward compatibility with existing sequence-based usage.

---

326-337: LGTM! Well-implemented random sampling with proper bounds checking.

The random sampling implementation:

• Correctly handles integer counts and fractional sampling
• Includes proper bounds checking (0 < case_parameters <= 1 for floats)
• Uses min() to prevent sampling more cases than available
• Appropriately uses random.sample for non-cryptographic case selection

The # nosec comments correctly indicate that non-cryptographic randomness is acceptable for test case sampling.

✨ Finishing Touches

• 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.