Fix N+1 query pattern in search title field fetching

[Copilot]

The search() function fetched title fields individually for each result using frappe.db.get_value() in a loop, generating N database queries for N results with title fields. For a typical search returning 20 results across 3 doctypes, this resulted in 20 queries instead of 3.

Changes

• Group results by doctype before title fetching to enable batching
• Batch fetch titles per doctype using frappe.get_all() with IN filter instead of individual get_value() calls
• Cache titles in lookup dict {doctype: {name: title}} for O(1) assignment during sorting
• Set limit_page_length=1000 to prevent unbounded queries while accommodating multi-word searches

Impact

Query count reduced from O(N) to O(M) where M is the number of unique doctypes with title fields. Typical improvement: 20 queries → 3 queries (85% reduction).

# Before: Individual query per result
for doctype in allowed_doctypes:
    for r in results:
        if r.doctype == doctype and r.rank > 0.0:
            meta = frappe.get_meta(r.doctype)
            if meta.title_field:
                r.title = frappe.db.get_value(r.doctype, r.name, meta.title_field)  # N queries

# After: Batch query per doctype
title_cache = {}
for dt, dt_results in results_by_doctype.items():
    meta = frappe.get_meta(dt)
    if meta.title_field:
        names = [r.name for r in dt_results]
        title_data = frappe.get_all(dt, filters={"name": ["in", names]}, 
                                    fields=["name", meta.title_field])  # M queries
        title_cache[dt] = {d.name: d.get(meta.title_field) for d in title_data}

Original prompt

---

This section details on the original issue you should resolve

<issue_title>N+1 Query Pattern in Title Field Fetching</issue_title>
<issue_description>

Metadata

• File(s): frappe_search/api/search.py:363-376
• Category: Database / API
• Severity: Medium
• Effort to Fix: Medium
• Estimated Performance Gain: 30-50%

Problem Description

The search() function fetches the title field value for each search result individually using frappe.db.get_value() inside a nested loop. With 20 results across multiple doctypes, this generates 20+ additional queries per search.

Impact Analysis

• CPU Impact: Low - Simple field lookups
• Memory Impact: Low
• Database Impact: Medium - One query per result with a title field
• User Experience Impact: Medium - Adds latency to search response
• Scalability Impact: Linear degradation with result count

Code Location

# File: frappe_search/api/search.py:362-376
# Sort results based on allowed_doctype's priority
for doctype in allowed_doctypes:
    for r in results:
        if r.doctype == doctype and r.rank > 0.0:
            try:
                meta = frappe.get_meta(r.doctype)  # Cached, OK
                if meta.title_field:
                    r.title = frappe.db.get_value(  # N+1 query per result
                        r.doctype, r.name, meta.title_field
                    )
            except Exception:
                frappe.clear_messages()

            sorted_results.append(r)

return sorted_results

Root Cause

The loop structure iterates through allowed_doctypes and then through all results, fetching the title field value individually for each matching result. While frappe.get_meta() is cached, frappe.db.get_value() is not and executes a query per call.

Proposed Solution

Batch fetch title fields by grouping results per doctype and using a single query per doctype:

@frappe.whitelist()
@redis_cache(ttl=180)
def search(text, start=0, limit=20, doctype="", allowed_doctypes=[]):
    """Search for given text in __global_search"""
    from frappe.query_builder.functions import Match

    results = []
    sorted_results = []

    if not allowed_doctypes or (doctype and doctype not in allowed_doctypes):
        return []

    for word in set(text.split("&")):
        word = word.strip()
        if not word:
            continue

        global_search = frappe.qb.Table("__global_search")
        rank = Match(global_search.content).Against(word)
        query = (
            frappe.qb.from_(global_search)
            .select(
                global_search.doctype,
                global_search.name,
                global_search.content,
                rank.as_("rank"),
            )
            .where(rank)
            .orderby("rank", order=frappe.qb.desc)
            .limit(limit)
        )

        if doctype:
            query = query.where(global_search.doctype == doctype)
        else:
            query = query.where(global_search.doctype.isin(allowed_doctypes))

        if cint(start) > 0:
            query = query.offset(start)

        result = query.run(as_dict=True)
        results.extend(result)

    # Group results by doctype for batch title fetching
    results_by_doctype = {}
    for r in results:
        if r.rank > 0.0:
            if r.doctype not in results_by_doctype:
                results_by_doctype[r.doctype] = []
            results_by_doctype[r.doctype].append(r)

    # Batch fetch titles per doctype
    title_cache = {}  # {doctype: {name: title}}
    for dt, dt_results in results_by_doctype.items():
        try:
            meta = frappe.get_meta(dt)
            if meta.title_field:
                names = [r.name for r in dt_results]
                title_data = frappe.get_all(
                    dt,
                    filters={"name": ["in", names]},
                    fields=["name", meta.title_field],
                    limit_page_length=0
                )
                title_cache[dt] = {d.name: d.get(meta.title_field) for d in title_data}
        except Exception:
            frappe.clear_messages()

    # Sort results based on allowed_doctype's priority and assign titles
    for dt in allowed_doctypes:
        for r in results:
            if r.doctype == dt and r.rank > 0.0:
                # Assign title from cache
                if dt in title_cache and r.name in title_cache[dt]:
                    r.title = title_cache[dt][r.name]
                sorted_results.append(r)

    return sorted_results

Implementation Steps

1. After fetching search results, group them by doctype
2. For each doctype with a title_field, batch fetch all titles in one query
3. Build a lookup dict {doctype: {name: title}}
4. In the sorting loop, assign titles from the cache instead of querying individually
5. Handle missing titles gracefully (document may have been deleted)

Testing Recommendations

• Verify titles are correctly displayed ...

• Fixes N+1 Query Pattern in Title Field Fetching #4

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.