[Security assistant] Conversation pagination patch

[stephmilovic]

Summary

Users with over 20 conversations will experience this error:

In the long term we need to implement pagination for the Assistant. Unfortunately, much of our conversation update logic on the frontend requires fetching all conversations. While a larger refactor is needed, this patch increases the fetch limit from 20 to 5000 conversations as a temporary fix.

UI query optimizations made to support 5000 conversations

1. useFetchCurrentUserConversations Hook:
   Updated to accept fields and filter arguments. We're now retrieving only 3-4 fields per conversation and excluding the messages field to reduce data load.

2. useAssistantOverlay Update:
   Previously, the hook fetched all conversations and mapped them to check for a title match. Now, it filters by title: ${title}, retrieving only the relevant conversation instead of fetching up to 5000.

3. Optimization in security_solution/public/assistant/provider.tsx:
   Instead of fetching all conversations to check if any exist, getUserConversations was replaced with getUserConversationsExist, optimizing the query to return a count only, reducing unnecessary data fetching.

4. Remove conversations fetch in in security_solution/public/assistant/stack_management/management_settings.tsx:
   I discovered a call to fetch all conversations only to locate the Welcome conversation, which was then passed to ConversationSettingsManagement in kbn-elastic-assistant, where all conversations were fetched again. This redundant call was removed, and the export from kbn-elastic-assistant was eliminated, as the references are now entirely internal.

Extra protection against Welcome conversation exists error

In the case the user has more than 5000 conversations, to prevent the "Welcome conversation exists" error and similar issues, I've added a sortField of is_default. This ensures that default conversations are returned first, followed by a secondary sort on updated_at. In the conversation selector side nav, conversations remain sorted by updated_at.

Previously, conversation.isDefault could be undefined, but it now defaults to false. This change ensures that conversations without isDefault are sorted correctly—without this, new conversations could be pushed to the bottom, even if they were recently updated. We rely on the updated_at field to display the most recent conversations at the top.

I did not modify the Elasticsearch type to require is_default, as legacy conversations won’t have this field. Instead, the value is defaulted to false in the API.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

⏳ Build in-progress

• Buildkite Build
• Commit: 2d0a5de
• Cloud Deployment
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-196782-2d0a5defe329
• Security Deployment

History

• 💔 Build #244028 failed ef961a5
• 💔 Build #244010 failed 56e8b5e
• 💔 Build #243680 failed e87233e