Add known fields for the Wazuh states index patterns creation

[asteriscos]

Description

Currently, the creation of the alerts index pattern uses a list of known fields to make sure that all the fields used in visualizations and tables are included in the creation of the index pattern. We need to add the known fields of the Wazuh states index patterns.

Tasks

• Implement known fields for the creation of all the Wazuh states index patterns.

Related code

https://github.com/wazuh/wazuh-dashboard-plugins/blob/4.14.0/plugins/main/public/utils/known-fields.js

[felipegonzalezmv]

Known Fields Migration: Hardcoded to Generated

Summary

This document explains the differences between the old hardcoded known fields and the new auto-generated known fields from official Wazuh templates.

Field Count Comparison

Index Type	Old (Hardcoded)	New (Generated)	Difference	Change
Alerts	444 fields	621 fields	+177	+39.9%
Monitoring	27 fields	12 fields	-15	-55.6%
Statistics	~75 fields	75 fields	0	0%

Alerts: New Fields Added (+177 fields)

Source of New Fields

The new fields come directly from the official Wazuh template:

https://raw.githubusercontent.com/wazuh/wazuh/4.14.0/extensions/elasticsearch/7.x/wazuh-template.json

1. Microsoft Graph API Fields (+172 fields)

The official Wazuh template includes extensive Microsoft Graph API fields that were not in the hardcoded version:

• data.ms-graph.* - 172 fields total
• Includes device management, audit events, security alerts, etc.
• These are part of the official wazuh-template.json but were missing from hardcoded fields

Examples:

data.ms-graph.configurationManagerClientEnabledFeatures (nested)
data.ms-graph.configurationManagerClientEnabledFeatures.compliancePolicy
data.ms-graph.configurationManagerClientEnabledFeatures.deviceConfiguration
data.ms-graph.deviceHealthAttestationState (nested)
data.ms-graph.deviceHealthAttestationState.secureBoot
data.ms-graph.resources (nested)
data.ms-graph.resources.modifiedProperties (nested)
... and 165 more ms-graph fields

2. Other New Fields (+5 fields)

• data.azureSignInStatus.* - 3 fields (Azure authentication)
• data.YARA.* - Additional YARA fields
• data.os.display_version - New in Wazuh 4.14.0
• data.parameters.extra_args - Process parameters

Verification: All Old Fields Are Present ✅

We verified that 100% of the old hardcoded fields (444) are present in the new generated fields (621). No fields were removed.

Monitoring: Fields Removed (-15 fields)

Official Template

The official monitoring template only defines 7 basic properties:

• Source: monitoring-template.ts
• Fields: timestamp, status, ip, host, name, id, cluster.name
• Total: 12 fields (including 5 meta fields: _id, _index, _score, _source, _type)

Fields Not in Official Template (15 fields removed)

The following fields were in the hardcoded version but are NOT in the official monitoring template:

Agent metadata fields:

• dateAdd - Agent registration date
• lastKeepAlive - Last agent communication
• mergedSum - Merged.mg checksum
• configSum - Configuration checksum
• node_name - Cluster node name
• manager - Manager name
• version - Agent version
• group - Agent groups

OS fields:

• os.arch, os.codename, os.major, os.name
• os.platform, os.uname, os.version

Investigation Findings

Where these fields come from:

• These fields are NOT defined in the official monitoring template
• They come from the Wazuh API response (GET /agents)
• The monitoring code inserts full agentInfo object (see server/start/monitoring/index.ts lines 238-250)
• They may be indexed dynamically (dynamic mapping) but not explicitly defined in template

Where these fields are used:

• Primarily used in agent tables and group overview components
• NOT used in monitoring/statistics dashboards or visualizations
• Statistics visualizations only use fields like analysisd.*, remoted.*, timestamp (which ARE in the template)

Why the Discrepancy?

The hardcoded FieldsMonitoring appears to have mixed:

1. Monitoring index fields (from template) - timestamp, status, ip, etc.
2. Agent API response fields (not in template) - dateAdd, lastKeepAlive, etc.

This suggests the hardcoded fields were created to cover both monitoring data AND agent data, even though agent fields may not be formally mapped in the monitoring template.

Recommendations

For Alerts: Use New Generated Fields (621) ✅

Reasons:

• ✅ Contains all 444 old fields (100% backward compatible)
• ✅ Adds official MS Graph integration fields
• ✅ Represents the official Wazuh template (source of truth)
• ✅ No breaking changes for existing visualizations

For Monitoring: Further Investigation Needed ⚠️

Options:

Option 1: Use generated fields (12) - If agent fields are truly obsolete

• Pro: Matches official template
• Con: May break agent tables if they query monitoring index

Option 2: Keep old fields (27) - If agent fields are actively used

• Pro: Maintains compatibility
• Con: Includes fields not in official template

Option 3: Hybrid approach - Use generated + add dynamic fields

• Pro: Best of both worlds
• Con: More complex

Next step: Verify if monitoring index is actually queried for agent fields like dateAdd, lastKeepAlive in production, or if these are only used when querying the Wazuh API directly.

Why More/Less Fields?

Alerts (+177 fields)

1. Template Completeness: Official template has more fields than manually maintained
2. Nested Fields Support: New generator correctly processes nested types with sub-properties
3. Microsoft Graph Integration: MS Graph API support added to Wazuh but not to hardcoded fields
4. Automatic Updates: New fields from template automatically included

Monitoring (-15 fields)

1. Template Simplicity: Official monitoring template only defines basic fields
2. Dynamic Mapping: Agent fields may be indexed dynamically without explicit mapping
3. Mixed Concerns: Hardcoded fields mixed monitoring data with agent API fields
4. Source Confusion: Fields from GET /agents API response vs actual index fields

Benefits of Generated Approach

1. ✅ Always in sync with official templates
2. ✅ No manual maintenance required
3. ✅ No missing fields from templates
4. ✅ Proper nested support
5. ✅ Version-aware (uses package.json version)
6. ✅ Consistent across datasets

Backward Compatibility

Alerts

• ✅ All 444 old fields are present
• ✅ No breaking changes for existing visualizations
• ✅ New fields are additive only
• ✅ Index patterns will have more fields available

Monitoring

• ⚠️ 15 fields removed (not in official template)
• ⚠️ Need to verify if these are used in production
• ⚠️ May need to add them back if actively used

Source of Truth

All generated fields come directly from official templates:

• Alerts: https://github.com/wazuh/wazuh/blob/4.14.0/extensions/elasticsearch/7.x/wazuh-template.json
• Monitoring: https://github.com/wazuh/wazuh-dashboard-plugins/blob/4.14.0/plugins/main/server/integration-files/monitoring-template.ts
• Statistics: https://github.com/wazuh/wazuh-dashboard-plugins/blob/4.14.0/plugins/main/server/integration-files/statistics-template.ts
• States (FIM, Inventory): https://github.com/wazuh/wazuh/blob/4.14.0/src/wazuh_modules/inventory_harvester/indexer/template/