[TENT] Add logs for troubleshooting

[alogfans]

Description

This PR improves the logging system for the TENT module to enhance observability and troubleshooting capabilities while reducing log noise.

1. Unified Log Rate Limiting Mechanism

• Created LogRateLimiter utility class (tent/src/common/log_rate_limiter.h)
• Replaced manual timestamp checking code with a clean, reusable API
• Provides predefined interval macros: LOG_RATE_LIMIT_1S, LOG_RATE_LIMIT_2S, LOG_RATE_LIMIT_5S, LOG_RATE_LIMIT_10S
• Before: 7 lines of repetitive code
• After: 3 lines with clear semantics

2. Proxy Manager (tent/src/runtime/proxy_manager.cpp)

• Added comprehensive logging for worker lifecycle, task processing, and memory operations
• Implemented rate limiting for frequent errors (buffer exhaustion, task failures)
• Removed verbose VLOG logs that add little value for troubleshooting
• All normal operations now use VLOG(1) - silent by default

3. RDMA Workers (tent/src/transport/rdma/workers.cpp)

• Enhanced WC (Work Completion) error diagnostics - most impactful change
  • Added getWcErrorDiagnostic() function providing actionable guidance for each error type
  • Enhanced log context: includes qp_num, target_id, length, and diagnostic hints
  • Example: "WC error: remote access error | Remote access error - peer denied access. Check remote memory permissions."
• Unified all rate limiting using LogRateLimiter
• Replaced LOG_EVERY_N with time-based rate limiting

4. Control Plane (tent/src/runtime/control_plane.cpp)

• Removed non-essential VLOG logs (initialization, startup success)
• Added critical error logs for:
  • SendData/RecvData buffer not found (data loss scenarios)
  • PinStageBuffer/UnpinStageBuffer failures (resource exhaustion)
  • Exception handling with request content preview (first 200 chars)
• Enhanced Delegate failure logs with target_id and opcode
• All new logs use rate limiting to prevent log spam

5. NVLink Transport (tent/src/transport/nvlink/nvlink_transport.cpp)

• Added rate limiting for CUDA errors
• Simplified log messages

Type of Change

• Types
  • Bug fix
  • New feature
    • Transfer Engine
    • Mooncake Store
    • Mooncake EP
    • Integration
    • P2P Store
    • Python Wheel
  • Breaking change
  • CI/CD
  • Documentation update
  • Other

How Has This Been Tested?

TBD

Checklist

• I have performed a self-review of my own code.
• I have formatted my own code using ./scripts/code_format.sh before submitting.
• I have updated the documentation.
• I have added tests to prove my changes are effective.

[gemini-code-assist]

Summary of Changes

Hello @alogfans, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly overhauls the logging infrastructure within the TENT module to improve system observability and streamline troubleshooting. The changes focus on reducing log noise by introducing a unified rate-limiting mechanism, enhancing error diagnostics with more contextual information, and ensuring critical failures are logged appropriately without overwhelming the system. The overall impact is a more informative and less noisy logging experience, making it easier to diagnose and resolve issues within the transfer engine.

Highlights

• Unified Log Rate Limiting: Introduced a new LogRateLimiter utility class and convenience macros (LOG_RATE_LIMIT_1S, etc.) to standardize and simplify rate-limiting of log messages across the TENT module, replacing repetitive manual timestamp checks.
• Enhanced RDMA Work Completion (WC) Error Diagnostics: Added a getWcErrorDiagnostic() function to provide actionable guidance for various RDMA WC error types, significantly improving troubleshooting capabilities for RDMA transport issues. Logs now include qp_num, target_id, length, and diagnostic hints.
• Improved Proxy Manager Logging: Added comprehensive logging for worker lifecycle events, task processing, and memory operations within the ProxyManager. Frequent errors like buffer exhaustion and task failures are now rate-limited to prevent log spam, and verbose VLOGs have been replaced with VLOG(1) for normal operations.
• Critical Error Logging in Control Plane: Implemented new critical error logs for data loss scenarios (SendData/RecvData buffer not found) and resource exhaustion (PinStageBuffer/UnpinStageBuffer failures). Exception handling for RPC calls now includes request content previews, and Delegate failure logs are enhanced with target_id and opcode, all with rate limiting.
• Refined NVLink and Metastore Logging: Applied rate limiting to CUDA error logs in the NVLink transport and simplified log messages. In the Metastore, LOG(FATAL) calls were downgraded to LOG(ERROR) for connection failures, and Redis error messages are now sanitized to prevent potential password leakage.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This is a great pull request that significantly improves the logging and observability of the TENT module. I particularly like the new LogRateLimiter for controlling log volume and the much-improved diagnostics for RDMA work completion errors—that will be incredibly helpful for debugging. Replacing LOG(FATAL) with LOG(ERROR) across the board is also a solid improvement for robustness.

I've found a critical compilation error due to a typo and a high-severity issue with the LogRateLimiter's handling of time that could lead to incorrect behavior. Once these are addressed, this PR will be in excellent shape. Great work on enhancing the system's maintainability!

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[staryxchen]

Glog provides several built-in macros for rate limiting purpose, like LOG_EVERY_N, LOG_FIRST_N and LOG_IF_EVERY_N. Can we reuse these macros?

[alogfans]

Glog provides several built-in macros for rate limiting purpose, like LOG_EVERY_N, LOG_FIRST_N and LOG_IF_EVERY_N. Can we reuse these macros?

We expect to avoid mass output in a short time period, so we prefer to throttle by wall-clock time rather than repeats (what these macros do)