Optimize conversation processing LLM costs (62% of total spend)

[beastoin]

Context

Fresh BQ export of users/{uid}/llm_usage shows the conversation processing pipeline consuming 62% of total LLM spend. Breakdown:

Feature	% of Total	Model
conv_action_items	19.8%	gpt-5.1
conv_structure	15.5%	gpt-5.1
other / untracked	15.5%	mixed
conversation_processing (legacy umbrella)	14.7%	gpt-5.1
conv_apps	12.2%	gpt-5.1

Root Cause Analysis

Every non-discarded conversation triggers 5-6 LLM calls, 3 of which hit gpt-5.1 with the full transcript:

Step	Feature	Model	Cost
1	conv_discard	gpt-4.1-mini	cheap
2	conv_structure	gpt-5.1	expensive
3	conv_action_items	gpt-5.1	expensive
4	conv_folder	gpt-4.1-mini	cheap
5	conv_apps (suggest)	gpt-4.1-mini	cheap
6	conv_apps (execute)	gpt-5.1	expensive

The same full transcript is sent to gpt-5.1 three separate times (structure, action_items, app_result).

Overuse paths

• sync.py:606 and postprocess_conversation.py:115 both call process_conversation(force_process=True) on already-processed conversations → 2-3x LLM cost for synced/postprocessed conversations
• _trigger_apps() runs get_suggested_apps_for_conversation() even when user has a preferred app set → wasted LLM call
• extract_action_items() fetches 50 recent action items for dedup context on every call → thousands of extra input tokens

Sub-issues (ordered by estimated impact)

See linked sub-issues below for individual optimization tasks.

Expected Outcome

Target: 25-30% reduction in conversation pipeline LLM costs without degrading output quality.

Data Source

BQ table: based-hardware:llm_usage.raw

Labels

intelligence, maintainer, p1

[beastoin]

Sub-issues (ordered by estimated impact)

#	Issue	Savings	Priority
1	#4636 — Merge conv_structure + conv_action_items into single LLM call	~15-20%	P1
2	#4637 — Downgrade conv_action_items to gpt-4.1-mini	~10-15%	P1
3	#4638 — conv_apps: use overview instead of full transcript	~5-8%	P2
4	#4639 — Skip conv_apps suggestion when user has preferred app	~2-3%	P2
5	#4640 — Cap action item dedup context from 50 to 10 items	~2-3%	P2
6	#4641 — Avoid full reprocess in sync and postprocess paths	Variable	P2

Combined target: 25-30% reduction in conversation pipeline LLM costs.

Note: #4636 and #4637 can't both apply fully (merging eliminates the separate action_items call). Recommended approach: implement #4636 first, then evaluate if the merged call still benefits from model downgrade (#4637).

[beastoin]

📝 The proposed solutions here are based on code analysis and have not been verified by the CTO. If you have a better approach to address this, feel free to propose it.

[beastoin]

Updated: Feb 5 BQ Data from @mon

Fresh llm_usage export confirms the cost concentration:

Feature	% of Calls	% of Cost	Model
conv_action_items	4.9%	24.1%	gpt-5.1
conversation_processing	7.0%	18.5%	gpt-5.1
conv_structure	4.9%	17.9%	gpt-5.1
conv_apps	9.8%	13.5%	gpt-5.1 + mini
other (untracked)	29.6%	13.2%	mostly mini
daily_summary	0.6%	4.5%	gpt-5.1

Model split: gpt-5.1 handles only 23.4% of calls but drives 86.4% of cost. gpt-4.1-mini handles 76.4% of calls at just 13.6% of cost.

Bottom line: 5 features on gpt-5.1 = 78.5% of cost with only 27.2% of calls. The conversation pipeline sub-features alone (action_items + structure + apps + processing) = 74% of total cost.

Note: Per-user breakdown still blocked by #4616 (missing uid field in llm_usage docs).