feat: Enhance LinkedIn stats granularity for better analytics

[teslashibe]

Enhance LinkedIn Stats Granularity for Better Analytics

📋 Current Situation

With the new LinkedIn profile fetching implementation (#116), we have two distinct operation types that are currently tracked under the same stats:

Current Stats (Combined Tracking)

LinkedInScrapes            // Both searchbyquery + getprofile operations
LinkedInProfiles           // Mixed: search results + individual detailed profiles
LinkedInErrors             // General errors
LinkedInAuthErrors         // Auth failures  
LinkedInRateErrors         // Rate limiting

Problem

• Can't distinguish between search operations vs profile fetch operations
• Can't track conversion rates (search → detailed profile fetch)
• Limited analytics insights for usage patterns
• Mixed profile counts (search results vs detailed profiles)

�� Proposed Solution

Add granular stats to distinguish between operation types and result types:

// Operation-specific counters
LinkedInSearches           // searchbyquery operations only
LinkedInProfileFetches     // getprofile operations only

// Result-specific counters  
LinkedInSearchProfiles     // profiles returned from searches
LinkedInDetailProfiles     // detailed profiles fetched

// Keep existing error stats (unchanged)
LinkedInErrors
LinkedInAuthErrors
LinkedInRateErrors

📊 Analytics Benefits

With granular stats, we could track:

{
  "linkedin_searches": 100,           // 100 search operations
  "linkedin_search_profiles": 500,    // 500 profiles found (5 avg per search)
  "linkedin_profile_fetches": 50,     // 50 detailed profile fetches  
  "linkedin_detail_profiles": 50,     // 50 detailed profiles extracted
}

Insights Enabled:

• Search→Fetch conversion rate: 50% (50 fetches / 100 searches)
• Average search results: 5 profiles per search
• Profile fetch success rate: 100%
• Usage patterns: Search-heavy vs fetch-heavy workloads

⚖️ Tradeoffs

Pros ✅	Cons ❌
Clear operation separation	More complex stats structure
Rich analytics capabilities	Breaking change to stats format
Conversion rate tracking	Additional code changes needed
Better usage insights	More stats to maintain

🤔 Questions for Team Discussion

1. Is the analytics value worth the added complexity?
2. Should we implement this now or defer to a future iteration?
3. Are there other stats patterns we should consider?
4. How would this impact existing monitoring/dashboards?

🛠 Implementation Impact

Files to Modify:

• internal/jobs/stats/stats.go - Add new stat types
• internal/jobs/linkedin.go - Update stat tracking calls
• internal/jobs/linkedin_test.go - Update test expectations

Backwards Compatibility:

• Current stats structure would change
• Existing monitoring may need updates
• Migration strategy needed for historical data

💭 Alternative Approaches

1. Keep current unified approach - Simple but limited analytics
2. Add metadata to existing stats - Hybrid approach with tagged metrics
3. Implement proposed granular stats - Full analytics capability

---

Please share your thoughts on whether we should implement granular LinkedIn stats or keep the current unified approach.

cc: @team-linkedin @analytics-team

[hide-on-bush-x]

@teslashibe LinkedInErrors is the sum of LinkedInAuthErrors and LinkedInRateErrors? or should we sum all them three for scoring?

Im thinking about LinkedInSearchProfiles and LinkedInDetailProfiles having different weights inside the linkedin pool but not sure if this makes sense, do this two operations have the same "complexity"?