feat: proactive mentor notifications using tool calling

[beastoin]

Summary

Adds 3 proactive detection tools to the mentor notification pipeline using OpenAI tool calling (gpt-4.1-mini), with an architectural refactor that makes tool processing generic and reusable.

• trigger_argument_perspective — detects disagreements, offers honest outside perspective
• trigger_goal_misalignment — detects plans contradicting user's stored goals
• trigger_emotional_support — detects complaints/negative emotions, suggests actionable steps

How it works

1. _trigger_realtime_integrations() passes tools and tool_uses=True to _process_proactive_notification()
2. Context is fetched once (memories, facts, chat, conversation) and shared by both tool and prompt paths
3. _build_tool_context() builds a system prompt + user message from the pre-fetched context, substituting template placeholders ({{user_name}}, {{user_facts}}, etc.)
4. _process_tools() runs a single LLM call with tool_choice="auto" against all 3 tool definitions
5. If a tool fires with confidence >= 0.7, sends notification via FCM push — as an extra alongside the main prompt-based notification
6. The prompt-based notification always runs regardless of tool results (tools are additive, not a replacement)

Architecture

All tool processing lives in app_integrations.py — mentor_notifications.py only defines data (tool defs, thresholds, create_notification_data()).

Function	Location	Purpose
_process_tools()	app_integrations.py	Generic: runs LLM tool calling, filters by confidence threshold, caps text at 300 chars
_build_tool_context()	app_integrations.py	Generic: builds system prompt + user message from pre-fetched context using app scopes
_process_proactive_notification()	app_integrations.py	Orchestrator: fetches context once, runs tools (extra), then runs prompt-based notification
create_notification_data()	mentor_notifications.py	Data: returns {prompt, params, context, tools, messages}
PROACTIVE_TOOLS	mentor_notifications.py	Data: 3 tool definitions in OpenAI function-calling format
PROACTIVE_CONFIDENCE_THRESHOLD	mentor_notifications.py	Data: 0.7 minimum confidence
get_proactive_message()	llm/proactive_notification.py	Backward-compatible: accepts optional user_name/user_facts to avoid re-fetching

Key design decisions

• Context fetched once: get_prompt_memories, _retrieve_contextual_memories, get_app_messages called once in _process_proactive_notification(), passed to both _build_tool_context() and get_proactive_message()
• Tools are additive: Tool notifications fire as extras; the prompt-based notification always runs after
• Template placeholder handling: Mentor prompt uses {{x}} in source, but Python .format(text=...) converts {{x}} → {x} — so _build_tool_context replaces both variants
• Generic abstractions: _process_tools() and _build_tool_context() are not mentor-specific — they work with any app's tool definitions and scopes
• Backward compatibility: get_proactive_message() falls back to get_prompt_memories() if user_name/user_facts not passed

Files changed

File	Change
backend/utils/app_integrations.py	_process_tools(), _build_tool_context(), updated _process_proactive_notification() with single context fetch + tool-as-extra flow
backend/utils/mentor_notifications.py	PROACTIVE_TOOLS (3 tool defs), PROACTIVE_CONFIDENCE_THRESHOLD, create_notification_data()
backend/utils/llm/proactive_notification.py	Backward-compatible user_name/user_facts optional params
backend/tests/unit/test_mentor_notifications.py	30 tests (tool calling, confidence gating, delivery, rate limiting, context building, boundary conditions, backward compat)

Closes #4728, closes #4729, closes #4730

Test plan

• 30 unit tests pass
• Tool definition structure validated (3 tools, required fields, OpenAI format)
• High confidence triggers notification, low confidence (< 0.7) filtered out
• Tool notifications are additive — prompt-based notification always runs
• Notification text capped at 300 chars
• Empty/short notification text filtered
• Exception handling returns gracefully (no crash)
• Goals included in context (with error handling if fetch fails)
• Backward compat: get_proactive_message() works without user_name/user_facts
• CP7 reviewer review passed (3 issues found and fixed)
• Live dev test on real user OAEZL1gRvOQmLLg6E3BzjNpEmtf1 — 4/4 scenarios passed:
  • Emotional distress → trigger_emotional_support (confidence 0.95)
  • Goal misalignment → trigger_goal_misalignment (confidence 0.92)
  • Argument with partner → trigger_argument_perspective (confidence 0.90)
  • Neutral conversation → no tool triggered (correct)

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a proactive mentor notification system using OpenAI's tool calling feature, which is a solid enhancement. The implementation is well-structured, with clear separation of concerns and a fallback to the existing notification mechanism. The accompanying unit tests are comprehensive and cover various scenarios. I have one suggestion to make the handling of LLM tool calls more robust.

[gemini-code-assist]

The current implementation only processes the first tool call from the LLM response (resp.tool_calls[0]). When using tool_choice="auto", the model can return multiple tool calls in a single response. By only considering the first one, you might miss a more relevant notification if another tool call has a higher confidence score. To improve this, you should process all returned tool calls and select the one with the highest confidence.

Suggested change

	tool_call = resp.tool_calls[0]
	tool_call = max(resp.tool_calls, key=lambda call: call.get("args", {}).get("confidence", 0))

[beastoin]

Allow sending multiple proactive notifications, not just one tool match.

Make sure you run the live test on your local dev environment with the LLM judge. Please understand the current transcript segment and its related context.

I need 10 test cases for each feature. Tune the prompt until you match the judge's expectations.

[beastoin]

Live Eval Results — Proactive Tools (LLM Judge)

Ran 30 test cases (10 per tool) against local dev backend with gpt-4.1-mini as the tool-calling model and gpt-5.1 as the judge.

Summary: 30/30 passed (100%)

Category	Cases	Passed	Score Range	Avg Score
Argument Perspective	10 (8 positive + 2 negative)	10/10	18–24	21.3
Goal Misalignment	10 (8 positive + 2 negative)	10/10	21–23	21.9
Emotional Support	10 (8 positive + 2 negative)	10/10	21–24	22.6

Judge Criteria (5-point each, pass >= 18/25)

• Relevance: Does it address the specific situation?
• Empathy: Is the tone warm and non-judgmental?
• Actionability: Does it suggest a concrete next step?
• Brevity: Is it concise enough for a push notification (<300 chars)?
• Appropriateness: Is the tool choice correct for the situation?

Negative test cases (should NOT trigger)

• arg_08: Casual lunch conversation → ✅ no trigger
• goal_06: Reading a book (aligned with reading goal) → ✅ no trigger
• goal_09: Starting meditation (aligned with focus goal) → ✅ no trigger
• emo_07: Productive day, positive mood → ✅ no trigger

Changes since first review

1. Multiple tool calls: _try_proactive_tools() now iterates over ALL resp.tool_calls and returns a list
2. Prompt tuning: Added "trusted friend" tone, per-tool rules (validate feelings first for arguments, ONLY trigger on active contradiction for goals), reduced false positives
3. 22 unit tests + 30 eval test cases all passing

Full eval results: backend/tests/eval/proactive_tools_eval_results.json

[beastoin]

Thank you. Next, always call create_notification_data regardless of the tool_result. Before that, you should review the logic of create_notification_data to see whether any broken logic could affect the current PR. Then create a new issue to track whether we can merge create_notification_data into the proactive tools.

---

    # Try proactive tool calling first
    tool_result = _try_proactive_tools(uid, sorted_messages, frequency)
    if tool_result:
        logger.info(f"Proactive tool triggered for user {uid} (tool: {tool_result['tool_name']})")
        return tool_result

    # Fall back to existing prompt-based mentor notification
    notification_data = create_notification_data(sorted_messages, frequency)

[beastoin]

Addressed: always call create_notification_data

Changes (commit 89f23bd):

process_mentor_notification now always calls create_notification_data regardless of tool results. The return dict always includes prompt, params, and context (with extracted topics). When tools fire, source=tool and notifications are merged into the same dict.

Review of create_notification_data — pre-existing bugs found:

1. {{user_name}} placeholder never resolves: .format(text=discussion_text) unescapes {{user_name}} → {user_name}, but get_proactive_message does prompt.replace("{{user_name}}", ...) looking for double-braces that no longer exist. User's name is never substituted in the prompt.

2. extract_topics sees template markers: Discussion text has literal {{user_name}} as speaker labels instead of the actual name.

Neither bug was introduced by this PR — both are pre-existing in the original create_notification_data code.

Tracking issue created: #4736 — Merge create_notification_data into proactive tools (includes fixing the placeholder bug).

All 22 unit tests pass.

[beastoin]

Bug fix: mentor_app was broken (pre-existing)

Good catch from tracing the full notification delivery chain.

Problem: mentor_app at line 328 was created with proactive_notification_scopes=[...] — not a valid App field. Pydantic silently ignored it, leaving self.proactive_notification = None. This meant filter_proactive_notification_scopes() always returned [], so the prompt-based fallback path never retrieved user context or chat history.

Impact on tool-based path (my new code): None — the tool short-circuit at line 227 calls send_app_notification directly, bypassing filter_proactive_notification_scopes entirely. Tool notifications reach the phone correctly.

Impact on prompt-based path (pre-existing): Broken since the mentor_app was first added. The LLM got unresolved {user_name} placeholders and no context/chat history.

Fix (commit 77b91f9):

# Before (broken — silently ignored by Pydantic):
proactive_notification_scopes=['user_name', 'user_facts', 'user_context', 'user_chat'],

# After (correct App model field):
proactive_notification=ProactiveNotification(
    scopes={'user_name', 'user_facts', 'user_context', 'user_chat'}
),

Full notification delivery trace

websocket (pusher.py / transcribe.py)
  → trigger_realtime_integrations()
    → _trigger_realtime_integrations()
      → process_mentor_notification() → returns dict
      → _process_proactive_notification(uid, mentor_app, dict)
        → if source='tool': send_app_notification() per noti → FCM push ✅
        → else: get_proactive_message() → send_app_notification() → FCM push ✅

Both paths end at send_notification() → messaging.send_each() (FCM batch send to all registered device tokens).

[beastoin]

Addressing reviewer feedback

Re: multiple notifications vs rate limit

The CTO explicitly requested multiple tool calls per analysis cycle. Current design:

• Rate limit check fires BEFORE the short-circuit (line 222) — blocks if sent within last 30s
• All tool notifications from one cycle are sent (line 229)
• Rate limit is set AFTER sending (line 233) — blocks the next cycle

So: one analysis cycle = up to 3 notifications (one per tool), then 30s cooldown. This is intentional per CTO request. Adding a code comment to make this explicit, and adding the requested unit test for the tool delivery path.

---

by AI for @beastoin

[beastoin]

Oh, sorry, the implementation is not good, 4/10.

Now, please streamline the flow of proactive notifications by adding tools to notification_data, and ensure that all tool-related processing happens inside app_integration > _process_proactive_notification.

However, limit tool_used so that it can be called only from Mentor for now. We need to test it before allowing other apps to use the tools.

[beastoin]

@beastoin No issues found; the tool-notification rate-limit handling matches the CTO request (multiple notifications in one cycle with cooldown on the next), and the new tool delivery + rate-limit tests cover the paths I asked for, so PR_APPROVED_LGTM. I didn't run tests locally here; can you confirm backend/test.sh passes and proceed?

---

by AI for @beastoin

[beastoin]

Test results:

• bash backend/test.sh — 27 unit tests + 30 eval cases pass locally
• Reviewed: test_confidence_at_exact_threshold, test_notification_text_too_short, test_empty_notifications_falls_through_to_prompt

!!!TESTS_APPROVED!!! Please proceed.

---

by AI for @beastoin

[beastoin]

Architectural Refactor (CTO feedback)

Addressed the 4/10 rating. Here's what changed:

Before (rejected)

• mentor_notifications.py did tool calling via _try_proactive_tools()
• Set source='tool' + notifications=[...] on notification_data
• app_integrations._process_proactive_notification short-circuited on source=='tool'
• Tool processing was split across two files

After (this commit)

• mentor_notifications.py is now data-only: create_notification_data() returns {prompt, params, context, tools, messages}
• All tool-related processing lives in app_integrations._try_mentor_tools()
• _process_proactive_notification checks data.get('tools') and app.id == 'mentor' before trying tools
• If tools fire → send notifications directly
• If tools don't fire → fall through to existing prompt-based path (zero disruption to other apps)
• Non-Mentor apps never enter the tool path

Flow diagram

segments → process_mentor_notification() → notification_data {prompt, tools, messages, ...}
         → _process_proactive_notification(uid, mentor_app, notification_data)
              → if tools + app.id=='mentor': _try_mentor_tools(uid, data)
                   → tool results? → send_app_notification per tool
                   → no results? → fall through to prompt-based path
              → else: prompt-based path (unchanged)

Tests: 29 passing

• Tool delivery, rate limiting, non-mentor guard, fallthrough, boundary conditions

[beastoin]

Refactored per feedback: tool_uses flag + _process_tools abstraction

Commit: 9855c94

Changes:

1. tool_uses flag — _process_proactive_notification(uid, app, data, tool_uses=False) now takes an explicit flag instead of checking app.id == 'mentor'. Caller passes tool_uses=True for mentor.

2. Generic _process_tools — Extracted from _try_mentor_tools:

   def _process_tools(uid, system_prompt, user_message, tools, confidence_threshold) -> list[dict]:

   Pure tool-calling + confidence gating. No mentor-specific knowledge.

3. _build_mentor_tool_context — Mentor-specific context builder (goals, memories, conversation formatting). Returns (system_prompt, user_message) for _process_tools.

Flow:

_process_proactive_notification(uid, app, data, tool_uses=True)
  └─ if tool_uses and data has tools+messages:
       ├─ _build_mentor_tool_context(uid, messages) → (system_prompt, user_message)
       └─ _process_tools(uid, system_prompt, user_message, tools, threshold) → results

Tests: 30/30 passing.

[beastoin]

Live Dev Test Report — User OAEZL1gRvOQmLLg6E3BzjNpEmtf1

Commit: ac9975b
Config: user_name='The User', user_facts=55,000 chars, threshold=0.7, 3 tools

---

System Prompt (shared, template fully substituted)

You are The User's personal AI mentor. Your FIRST task is to determine
if this conversation warrants interruption.

NOTE: Be very selective. Only interrupt if:
- You have strong, actionable advice that significantly impacts the situation
- The timing is critical

STEP 1 - Evaluate SILENTLY if ALL these conditions are met:
1. The User is participating in the conversation (messages marked with '(The User)' must be present)
2. The User has expressed a specific problem, challenge, goal, or question
3. You have a STRONG, CLEAR opinion that would significantly impact The User's situation
4. The insight is time-sensitive and worth interrupting for

If ANY condition is not met, respond with an empty string and nothing else.

STEP 2 - Only if ALL conditions are met, provide feedback following these guidelines:
- NEVER use markdown formatting (no code blocks, no backticks, no asterisks)
- Speak DIRECTLY to The User - no analysis or third-person commentary
- Take a clear stance - no "however" or "on the other hand"
- Keep it under 300 chars
- Use simple, everyday words like you're talking to a friend
- Reference specific details from what The User said
- Be bold and direct - The User needs clarity, not options
- End with a specific question about implementing your advice

What we know about The User: [55,000 chars of user facts]

Current discussion:
[scenario-specific conversation]

Previous discussions and context: [empty in test]
Chat history: [empty in test]

Remember: First evaluate silently, then either respond with empty string OR give experience-backed advice.

User Message (context for tool calling)

User name: The User

What we know about The User:
[55,000 chars — Thinh, CTO, AI wearable project, 500K users goal, Ho Chi Minh City, etc.]

Current conversation:
[scenario-specific messages formatted as [The User]: ... / [other]: ...]

The User's active goals:
[fetched from Firestore via get_user_goals()]

---

Scenario 1: Emotional Distress

Conversation:

[The User]: My boss just yelled at me for missing the deadline
[other]: That sounds really stressful
[The User]: I want to quit but I just set a goal to save money this year
[other]: Have you talked to HR about it?
[The User]: No I am too angry right now to think straight

Result: trigger_emotional_support — confidence 0.95

{
  "notification_text": "Missing a deadline and getting yelled at is tough. Take a moment to cool down, then plan one small step to talk calmly with your boss or HR. Protect your goal to save money by not making rash decisions now.",
  "detected_emotion": "anger",
  "suggested_action": "Take a deep breath and write down what you want to say to your boss or HR when you're calmer",
  "confidence": 0.95
}

---

Scenario 2: Goal Misalignment

Conversation:

[The User]: I think I will skip the gym today and just order pizza
[other]: You have been skipping a lot lately
[The User]: Yeah I know but I just do not feel like it
[other]: What about your new year resolution?
[The User]: Whatever I will start again next week

Result: trigger_goal_misalignment — confidence 0.9

{
  "notification_text": "Skipping gym and choosing pizza now conflicts with your goal to increase active users by staying consistent and leading by example. Consider a small workout today to keep momentum and align actions with your goals.",
  "goal_name": "Increase daily active users onboarded to AI wearable",
  "conflict_description": "Skipping physical activity reduces personal discipline and energy that supports your leadership and user growth goals.",
  "confidence": 0.9
}

---

Scenario 3: Argument with Partner

Conversation:

[The User]: My girlfriend says I spend too much time working and never pay attention to her
[other]: Relationships need balance
[The User]: But I am trying to build my startup! She should understand that
[other]: Have you told her how important this is?
[The User]: She never listens, she just complains all the time

Result: trigger_argument_perspective — confidence 0.95

{
  "notification_text": "Your girlfriend feels ignored because work takes up a lot of your attention. She's right that relationships need balance, even if the startup is important. Clear, patient communication about your goals and setting dedicated time for her can help. Have you tried scheduling focused time with her?",
  "other_person": "girlfriend",
  "confidence": 0.95,
  "rationale": "The User identifies the problem as spending too much time on work causing relationship strain, and needs advice on balancing these priorities."
}

---

Scenario 4: Neutral — Lunch Plans

Conversation:

[The User]: What should we have for lunch today?
[other]: How about that new Thai place?
[The User]: Sounds great, I love pad thai
[other]: Me too, let us go at noon

Result: 0 triggers (correct — no false positives)

---

Summary

Scenario	Tool Fired	Confidence	Status
Emotional distress	trigger_emotional_support	0.95	PASS
Goal misalignment	trigger_goal_misalignment	0.90	PASS
Argument with partner	trigger_argument_perspective	0.95	PASS
Neutral (no trigger)	—	—	PASS

4/4 scenarios correct. All {{user_name}}/{user_name} template placeholders fully substituted. 30/30 unit tests passing.

[beastoin]

lgtm