Add known fields for the wazuh states index patterns creation #7777
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced new constants for various Wazuh states index types in `constants.ts`. - Updated `saved-objects.js` to include handling for the new states index types. - Added detailed known fields for Wazuh states, including vulnerabilities, inventory systems, and FIM, in `known-fields.js`.
1 task
Desvelao
reviewed
Sep 26, 2025
Desvelao
reviewed
Sep 26, 2025
Desvelao
reviewed
Sep 26, 2025
Desvelao
reviewed
Sep 26, 2025
|
Add entry to the changelog |
- Replace 'Basic Elasticsearch fields' with 'Basic index fields' - Addresses review comment about removing Elasticsearch references Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Remove fallback to KnownFields (alerts fields) for states patterns - Throw specific error when no known fields are found for states pattern - Prevents mixing alerts fields with states index patterns - Addresses review comment about inappropriate fallback logic Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Replace multiple if/else statements with PATTERN_TO_INDEX_TYPE_MAP object - Use Object.keys().find() for cleaner pattern matching - Improve code readability and maintainability - Addresses review feedback about code elegance Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Restore original switch statement logic for states index types - Apply code formatting improvements to pattern mapping object - Ensure proper line breaks and indentation for better readability - Complete solution for passing correct indexType to known fields selection Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
Features: - Create generate-known-fields.js script to automatically fetch and convert Wazuh index templates to known fields format - Support vulnerabilities and alerts templates from official Wazuh repos - Generate JSON files with properly mapped field types and properties - Add known-fields-loader.js for dynamic loading of generated fields - Update saved-objects.js to use generated fields instead of hardcoded ones - Add comprehensive documentation in scripts/README.md Benefits: - Ensures known fields stay synchronized with official templates - Eliminates manual maintenance of field definitions - Reduces risk of field mismatches between server and dashboard - Provides foundation for supporting all states index patterns Generated: - plugins/main/public/utils/known-fields/alerts.json (614 fields) - plugins/main/public/utils/known-fields/states-vulnerabilities.json (52 fields) This addresses the automation requirement mentioned in review feedback to keep known fields synchronized with upstream template definitions. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Update known-fields-loader.js to export generated alerts fields as KnownFields - Remove dependency on manually maintained known-fields.js - Update all imports in saved-objects.js and test files to use loader - Apply code formatting improvements to generate-known-fields.js - Now all known fields come from automatically generated JSON files This completes the migration from hardcoded fields to generated fields, ensuring all field definitions stay synchronized with official templates. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
…erns - Add all inventory and FIM templates to generation script (14 new templates) - Generate JSON files for all states patterns: vulnerabilities, FIM, and inventory types - Update known-fields-loader with proper imports for all generated fields - Remove fallback dependencies and use specific fields for each pattern type Generated files include: - states-fim-files.json (24 fields) - states-fim-registries.json (30 fields) - states-inventory-system.json (29 fields) - states-inventory-hardware.json (21 fields) - states-inventory-networks.json (20 fields) - states-inventory-packages.json (26 fields) - states-inventory-ports.json (24 fields) - states-inventory-processes.json (23 fields) - states-inventory-protocols.json (18 fields) - states-inventory-users.json (44 fields) - states-inventory-groups.json (20 fields) - states-inventory-services.json (45 fields) - states-inventory-interfaces.json (27 fields) - states-inventory-hotfixes.json (14 fields) - states-inventory-browser-extensions.json (35 fields) This resolves the issue by ensuring all Wazuh states index patterns have proper known fields definitions, preventing visualization errors when indices don't exist yet. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Fix TypeScript template parsing with improved regex and safer evaluation - Generate monitoring.json (12 fields) from monitoring-template.ts - Generate statistics.json (75 fields) from statistics-template.ts - Update known-fields-loader to import monitoring and statistics fields - Now all Wazuh index patterns have automatically generated known fields Total coverage achieved: ✅ Alerts: 614 fields ✅ Monitoring: 12 fields ✅ Statistics: 75 fields ✅ States Vulnerabilities: 52 fields ✅ States FIM: 2 types (24-30 fields each) ✅ States Inventory: 13 types (14-45 fields each) This completes the full automation of known fields generation, ensuring 100% synchronization with official Wazuh templates and resolving all visualization errors when indices don't exist yet. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Create known-fields-loader.test.tsx with full test coverage - Test KnownFields exports and GeneratedKnownFields structure - Validate pattern extraction and field mapping functions - Test states-specific field validation for all types - Ensure proper field structure and metadata for all templates - Verify vulnerabilities package/vulnerability fields - Validate inventory system host/OS fields - Test monitoring timestamp/status fields - Verify statistics analysisd/remoted fields Tests cover all new functionality introduced by the automated known fields generation system. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Add entry for automated known fields generation system - Reference PR #7777 in changelog - Document feature in Wazuh v4.14.0 Added section This completes the documentation requirements for the comprehensive known fields automation implementation. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
…tion - Fix generate-known-fields.js to use absolute paths relative to script location - Prevent creation of duplicate directory structures when run from different locations - Add 'generate:known-fields' script to plugins/main/package.json - Now script works correctly whether executed from project root or plugins/main directory This ensures the known fields generation script can be run reliably from any location within the project structure. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Replace OpenSearch/Elasticsearch references with neutral 'index' terminology - Update mapOpenSearchType → mapIndexFieldType in generate-known-fields.js - Update comments to use 'index fields' instead of vendor-specific terms - Regenerate JSON files with updated function references - Maintain only necessary URL references to official repository paths This ensures vendor-neutral terminology while maintaining functionality. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Format all generated known fields JSON files according to project standards - Ensures consistent formatting for generated content - Only applies to files we created, not the entire project Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com>
- Apply prettier formatting to known-fields-loader.test.tsx - Ensure code style consistency across the project - Fix CI/CD formatting warnings
…o 7776-add-known-fields-for-the-wazuh-states-index-patterns-creation
…This change eliminates the reliance on hardcoded fields, streamlining the known fields management process. All known fields are now generated dynamically, ensuring synchronization with official templates.
This commit unifies the group data structure by consolidating descriptions, names, and user groups into a single format. The deprecated known fields reader has been removed, streamlining the sample data generation process. This change enhances maintainability and ensures consistency in the generated data.
This commit updates the ensureIndexPatternIsCreated function to accept an indexType parameter, allowing for more precise validation of index patterns. Additionally, various data source components have been modified to pass the appropriate indexType constants, improving maintainability and clarity in the codebase. Unused imports and deprecated functions have been removed to streamline the implementation.
…ons for Wazuh states index patterns and support
🟢 Create any Wazuh states index pattern when no underlying indices existScreen.Recording.2025-10-08.at.6.36.34.PM.mov🟢 Verify that the index pattern includes all expected fields specific to that type
Confirm visualizations and data tables work without field-related errorsRun `yarn test:jest` to verify all tests passExecute `node scripts/generate-known-fields.js` to test the generation script |
…-index-patterns-creation
…This change moves the field retrieval and update logic into a try block, improving error handling and maintainability of the code.
Comment on lines
+307
to
+347
| static getIndicesFields = async (pattern, indexType) => { | ||
| try { | ||
| const response = await GenericRequest.request( | ||
| //we check if indices exist before creating the index pattern | ||
| 'GET', | ||
| `/api/index_patterns/_fields_for_wildcard?pattern=${pattern}&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score`, | ||
| {}, | ||
| ); | ||
| return response.data.fields; | ||
| } catch (error) { | ||
| switch (indexType) { | ||
| case WAZUH_INDEX_TYPE_MONITORING: | ||
| return FieldsMonitoring; | ||
| case WAZUH_INDEX_TYPE_STATISTICS: | ||
| return FieldsStatistics; | ||
| case WAZUH_INDEX_TYPE_ALERTS: | ||
| return KnownFields; | ||
| case WAZUH_INDEX_TYPE_STATES_VULNERABILITIES: | ||
| case WAZUH_INDEX_TYPE_STATES_FIM_FILES: | ||
| case WAZUH_INDEX_TYPE_STATES_FIM_REGISTRIES: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_SYSTEM: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_HARDWARE: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_NETWORKS: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_PACKAGES: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_PORTS: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_PROCESSES: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_PROTOCOLS: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_USERS: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_GROUPS: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_SERVICES: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_INTERFACES: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_HOTFIXES: | ||
| case WAZUH_INDEX_TYPE_STATES_INVENTORY_BROWSER_EXTENSIONS: | ||
| // Get known fields by index type to ensure expected fields are returned | ||
| const statesFields = getKnownFieldsByIndexType(indexType); | ||
| if (statesFields) { | ||
| return statesFields; | ||
| } | ||
| // If no specific states fields found, throw error | ||
| const statesError = ErrorFactory.create(WarningError, { | ||
| error, | ||
| message: `No known fields defined for index type: ${indexType}`, | ||
| }); | ||
| throw statesError; |
There was a problem hiding this comment.
Keep missing-index detection intact
When _fields_for_wildcard throws (e.g. because the wazuh-states indices are missing), we now return the static statesFields, so SavedObject.getIndicesFields resolves successfully. Downstream, existsIndices treats every successful resolve as exist: true, which means screens such as the Vulnerability Detection guard and the Statistics overview will no longer surface the "index missing" prompts and will try to create/use index patterns even though no indices exist.
On clusters where those features are disabled (no indices yet), the UI regresses to an empty table instead of guiding the user. We still need to surface the "no indices" condition — either by re-throwing on 404 or signalling exist: false — while reusing the known-field list only for index-pattern creation once existence has been confirmed.
There was a problem hiding this comment.
You can apply this patch
diff --git a/plugins/main/public/react-services/check-index.ts b/plugins/main/public/react-services/check-index.ts
index 22e849846..17ffa6ddf 100644
--- a/plugins/main/public/react-services/check-index.ts
+++ b/plugins/main/public/react-services/check-index.ts
@@ -1,15 +1,36 @@
import { SavedObject } from '.';
import { getSavedObjects } from '../kibana-services';
-import { NOT_TIME_FIELD_NAME_INDEX_PATTERN } from '../../common/constants';
+import {
+ NOT_TIME_FIELD_NAME_INDEX_PATTERN,
+ WAZUH_INDEX_TYPE_STATISTICS,
+} from '../../common/constants';
+import { GenericRequest } from './generic-request';
+import { getKnownFieldsByIndexType } from '../utils/known-fields-loader';
export async function existsIndices(indexPatternId: string, indexType: string) {
try {
- const fields = await SavedObject.getIndicesFields(
- indexPatternId,
- indexType,
+ // Use Dashboards API to check if any indices match the wildcard.
+ // We request with `returnError=true` so we can distinguish 404s.
+ const response = await GenericRequest.request(
+ 'GET',
+ `/api/index_patterns/_fields_for_wildcard?pattern=${indexPatternId}&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score`,
+ {},
+ true,
);
- return { exist: true, fields };
- } catch (error) {
+
+ // If the request succeeded, indices exist. For states/statistics patterns,
+ // prefer known fields for index-pattern creation to ensure consistency.
+ const apiFields = response?.data?.fields ?? [];
+ const isStatesType = typeof indexType === 'string' && indexType.startsWith('states-');
+ const shouldUseKnownFields = isStatesType || indexType === WAZUH_INDEX_TYPE_STATISTICS;
+ const knownFields = shouldUseKnownFields
+ ? getKnownFieldsByIndexType(indexType) || undefined
+ : undefined;
+
+ return { exist: true, fields: knownFields ?? apiFields };
+ } catch (error: any) {
+ // Keep missing-index detection intact: any error means no indices for guards.
+ // In particular, a 404 from `_fields_for_wildcard` must yield `exist: false`.
return { exist: false };
}
}
🟢 TEST🟢 CR🟢 Create any Wazuh states index pattern when no underlying indices existbefore index pattern creation index pattern creation 🟢 Verify that the index pattern includes all expected fields specific to that type
🟢 Confirm visualizations and data tables work without field-related errors
🟢 Run `yarn test:jest` to verify all tests pass
🟡 Execute `node scripts/generate-known-fields.js` to test the generation script
|
…ions. The test for pattern not found now uses async/await syntax for clarity, and the test for index pattern field retrieval has been modified to ensure it does not throw errors when fields retrieval fails, reflecting the updated behavior of the select() method.
Comment on lines
+35
to
+40
| it('should not throw error when get fields for index pattern rejects', async () => { | ||
| // The catch block in select() silently catches errors from getFieldsForIndexPattern | ||
| patternService.getFieldsForIndexPattern.mockRejectedValue( | ||
| new Error('Fields error'), | ||
| ); | ||
| await expect(patternDataSource.select()).resolves.not.toThrow(); |
There was a problem hiding this comment.
Minor note: this test currently uses resolves.not.toThrow(), which doesn't actually verify what it seems.
toThrow() only works for synchronous exceptions, not for async promises. Since select() returns a promise, this matcher won't detect if it rejects.
You can fix it by either:
await expect(patternDataSource.select()).resolves.toBeUndefined();or, if you prefer to keep the “should not throw” semantics:
await expect(async () => {
await patternDataSource.select();
}).not.toThrow();This way the test will correctly fail if select() rejects or throws.
asteriscos
reviewed
Oct 8, 2025
Comment on lines
+85
to
+86
| 2. **Future Enhancement**: Could be integrated into CI/CD to run automatically on releases | ||
| 3. **Fallback**: Manual fields still available as backup in `known-fields.js` |
There was a problem hiding this comment.
Suggested change
| 2. **Future Enhancement**: Could be integrated into CI/CD to run automatically on releases | |
| 3. **Fallback**: Manual fields still available as backup in `known-fields.js` | |
| 2. **Fallback**: Manual fields still available as backup in `known-fields.js` |
Comment on lines
+97
to
+110
| ## Recent Improvements | ||
|
|
||
| - ✅ Dynamic version loading from package.json (no hardcoded versions) | ||
| - ✅ Proper nested field type handling | ||
| - ✅ Moved templates from TypeScript to JSON format for simplicity | ||
| - ✅ Generated files in public/utils/known-fields/ (accessible from both client and server via dynamic path resolution) | ||
| - ✅ Added support for all inventory state templates | ||
| - ✅ Monitoring fields intentionally excluded from auto-generation (template incomplete) | ||
|
|
||
| ## Future Improvements | ||
|
|
||
| - [ ] Integrate with CI/CD for automatic updates on template changes | ||
| - [ ] Add validation to ensure generated fields match expected format | ||
| - [ ] Add diffing tool to show changes between template versions |
There was a problem hiding this comment.
Suggested change
| ## Recent Improvements | |
| - ✅ Dynamic version loading from package.json (no hardcoded versions) | |
| - ✅ Proper nested field type handling | |
| - ✅ Moved templates from TypeScript to JSON format for simplicity | |
| - ✅ Generated files in public/utils/known-fields/ (accessible from both client and server via dynamic path resolution) | |
| - ✅ Added support for all inventory state templates | |
| - ✅ Monitoring fields intentionally excluded from auto-generation (template incomplete) | |
| ## Future Improvements | |
| - [ ] Integrate with CI/CD for automatic updates on template changes | |
| - [ ] Add validation to ensure generated fields match expected format | |
| - [ ] Add diffing tool to show changes between template versions |
gonzaarancibia
previously approved these changes
Oct 8, 2025
Comment on lines
+62
to
+68
| - `alerts.json` - Fields for alert index patterns (621 fields) | ||
| - ~~`monitoring.json`~~ - ⚠️ **NOT generated** (uses `monitoring-fields.ts` instead) | ||
| - `statistics.json` - Fields for statistics patterns (75 fields) | ||
| - `states-vulnerabilities.json` - Fields for vulnerability state patterns (52 fields) | ||
| - `states-fim-files.json` - FIM files patterns (24 fields) | ||
| - `states-fim-registries.json` - FIM registries patterns (30 fields) | ||
| - `states-inventory-*.json` - All inventory state patterns (system, hardware, networks, packages, ports, processes, protocols, users, groups, services, interfaces, hotfixes, browser-extensions) |
There was a problem hiding this comment.
I don’t think it makes sense to mention the number of existing fields. Because then it has to be maintained, whether manually or automatically, and it doesn’t add any value, in my opinion.
Comment on lines
+70
to
+74
| ### Why is monitoring NOT auto-generated? | ||
|
|
||
| The monitoring template (`monitoring-template.json`) only defines basic index-level fields like `timestamp`, `status`, `ip`, `host`, etc. However, the actual monitoring index contains additional agent-specific fields (`dateAdd`, `lastKeepAlive`, `mergedSum`, `configSum`, etc.) that are inserted dynamically by the Wazuh server and are not part of the index template. | ||
|
|
||
| To ensure compatibility, monitoring fields are maintained manually in `plugins/main/public/utils/monitoring-fields.ts`. |
There was a problem hiding this comment.
And couldn’t we add a special check for monitoring, that verifies the fields that do exist in the index template and aren’t inserted dynamically? At least so that part of it is checked, and the rest can be checked manually. It would be like a partial check. One part would be verified automatically, and the other manually.
🟢 Test Legend
🧪 UI Tests
📋Test Details🟢 1️⃣ Create any Wazuh states index pattern when no underlying indices exist
Before After 🟢 2️⃣ In **Wazuh Dashboard > Dashboard management > Dashboards management > Index patterns > [Wazuh states pattern details]**: Inspect available fields → Index pattern includes all expected fields for that type
🔴 3️⃣ In **Wazuh Dashboard > Visualizations / Data tables**: Load visuals using the Wazuh states pattern → Visualizations render without field-related errors
🟢 5️⃣ In **Wazuh dev scripts**: Execute `node scripts/generate-known-fields.js` → Script runs and generates known fields without errors
|
This commit introduces the WAZUH_INDEX_TYPE_STATES_INVENTORY constant and integrates it into various components, including the validation logic and known fields loader. A new JSON file for states inventory fields is created, consolidating relevant data for improved management and accessibility. Additionally, a script is added to generate combined inventory fields, enhancing the overall structure and maintainability of the codebase.
This commit enhances the clarity of the combined inventory fields generation by formatting the function call to `generateCombinedInventoryFields` across multiple lines. This change improves code readability and maintainability without altering the functionality.
|
|
|
|
I was running some tests, and it behaves differently from the Screen.Recording.2025-10-08.at.11.00.55.PM.movScreen.Recording.2025-10-08.at.10.49.00.PM.mov |
asteriscos
deleted the
7776-add-known-fields-for-the-wazuh-states-index-patterns-creation
branch
3 tasks
3 tasks
Desvelao
added a commit
that referenced
this pull request
Oct 27, 2025
* Add Wazuh states index types and known fields for states patterns - Introduced new constants for various Wazuh states index types in `constants.ts`. - Updated `saved-objects.js` to include handling for the new states index types. - Added detailed known fields for Wazuh states, including vulnerabilities, inventory systems, and FIM, in `known-fields.js`. * Remove Elasticsearch references from known fields comments - Replace 'Basic Elasticsearch fields' with 'Basic index fields' - Addresses review comment about removing Elasticsearch references Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Fix states patterns to avoid fallback to alerts KnownFields - Remove fallback to KnownFields (alerts fields) for states patterns - Throw specific error when no known fields are found for states pattern - Prevents mixing alerts fields with states index patterns - Addresses review comment about inappropriate fallback logic Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Refactor getIndexTypeFromPattern to use clean object mapping - Replace multiple if/else statements with PATTERN_TO_INDEX_TYPE_MAP object - Use Object.keys().find() for cleaner pattern matching - Improve code readability and maintainability - Addresses review feedback about code elegance Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Restore switch statement and apply code formatting improvements - Restore original switch statement logic for states index types - Apply code formatting improvements to pattern mapping object - Ensure proper line breaks and indentation for better readability - Complete solution for passing correct indexType to known fields selection Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Add automated known fields generation system Features: - Create generate-known-fields.js script to automatically fetch and convert Wazuh index templates to known fields format - Support vulnerabilities and alerts templates from official Wazuh repos - Generate JSON files with properly mapped field types and properties - Add known-fields-loader.js for dynamic loading of generated fields - Update saved-objects.js to use generated fields instead of hardcoded ones - Add comprehensive documentation in scripts/README.md Benefits: - Ensures known fields stay synchronized with official templates - Eliminates manual maintenance of field definitions - Reduces risk of field mismatches between server and dashboard - Provides foundation for supporting all states index patterns Generated: - plugins/main/public/utils/known-fields/alerts.json (614 fields) - plugins/main/public/utils/known-fields/states-vulnerabilities.json (52 fields) This addresses the automation requirement mentioned in review feedback to keep known fields synchronized with upstream template definitions. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Migrate to use generated JSON fields as primary source - Update known-fields-loader.js to export generated alerts fields as KnownFields - Remove dependency on manually maintained known-fields.js - Update all imports in saved-objects.js and test files to use loader - Apply code formatting improvements to generate-known-fields.js - Now all known fields come from automatically generated JSON files This completes the migration from hardcoded fields to generated fields, ensuring all field definitions stay synchronized with official templates. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Complete implementation of automated known fields for all states patterns - Add all inventory and FIM templates to generation script (14 new templates) - Generate JSON files for all states patterns: vulnerabilities, FIM, and inventory types - Update known-fields-loader with proper imports for all generated fields - Remove fallback dependencies and use specific fields for each pattern type Generated files include: - states-fim-files.json (24 fields) - states-fim-registries.json (30 fields) - states-inventory-system.json (29 fields) - states-inventory-hardware.json (21 fields) - states-inventory-networks.json (20 fields) - states-inventory-packages.json (26 fields) - states-inventory-ports.json (24 fields) - states-inventory-processes.json (23 fields) - states-inventory-protocols.json (18 fields) - states-inventory-users.json (44 fields) - states-inventory-groups.json (20 fields) - states-inventory-services.json (45 fields) - states-inventory-interfaces.json (27 fields) - states-inventory-hotfixes.json (14 fields) - states-inventory-browser-extensions.json (35 fields) This resolves the issue by ensuring all Wazuh states index patterns have proper known fields definitions, preventing visualization errors when indices don't exist yet. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Complete TypeScript parsing support and generate all remaining templates - Fix TypeScript template parsing with improved regex and safer evaluation - Generate monitoring.json (12 fields) from monitoring-template.ts - Generate statistics.json (75 fields) from statistics-template.ts - Update known-fields-loader to import monitoring and statistics fields - Now all Wazuh index patterns have automatically generated known fields Total coverage achieved: ✅ Alerts: 614 fields ✅ Monitoring: 12 fields ✅ Statistics: 75 fields ✅ States Vulnerabilities: 52 fields ✅ States FIM: 2 types (24-30 fields each) ✅ States Inventory: 13 types (14-45 fields each) This completes the full automation of known fields generation, ensuring 100% synchronization with official Wazuh templates and resolving all visualization errors when indices don't exist yet. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Add comprehensive tests for known-fields-loader functionality - Create known-fields-loader.test.tsx with full test coverage - Test KnownFields exports and GeneratedKnownFields structure - Validate pattern extraction and field mapping functions - Test states-specific field validation for all types - Ensure proper field structure and metadata for all templates - Verify vulnerabilities package/vulnerability fields - Validate inventory system host/OS fields - Test monitoring timestamp/status fields - Verify statistics analysisd/remoted fields Tests cover all new functionality introduced by the automated known fields generation system. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Update CHANGELOG.md with automated known fields feature - Add entry for automated known fields generation system - Reference PR #7777 in changelog - Document feature in Wazuh v4.14.0 Added section This completes the documentation requirements for the comprehensive known fields automation implementation. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Fix script path resolution and add npm script for known fields generation - Fix generate-known-fields.js to use absolute paths relative to script location - Prevent creation of duplicate directory structures when run from different locations - Add 'generate:known-fields' script to plugins/main/package.json - Now script works correctly whether executed from project root or plugins/main directory This ensures the known fields generation script can be run reliably from any location within the project structure. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Use neutral terminology instead of vendor-specific references - Replace OpenSearch/Elasticsearch references with neutral 'index' terminology - Update mapOpenSearchType → mapIndexFieldType in generate-known-fields.js - Update comments to use 'index fields' instead of vendor-specific terms - Regenerate JSON files with updated function references - Maintain only necessary URL references to official repository paths This ensures vendor-neutral terminology while maintaining functionality. Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * Apply Prettier formatting to generated JSON files - Format all generated known fields JSON files according to project standards - Ensures consistent formatting for generated content - Only applies to files we created, not the entire project Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> * fix: prettier formatting for known-fields-loader test file - Apply prettier formatting to known-fields-loader.test.tsx - Ensure code style consistency across the project - Fix CI/CD formatting warnings * Remove known-fields.js file containing predefined field definitions. This change eliminates the reliance on hardcoded fields, streamlining the known fields management process. All known fields are now generated dynamically, ensuring synchronization with official templates. * Integrate known fields with sample data generation for IT Hygiene Groups - Add known-fields-reader utility to load and process known fields JSON files - Update states-inventory-groups generator to use known fields as source of truth - Ensure perfect synchronization between field definitions and generated sample data - Maintain backward compatibility with fallback to legacy generation - Eliminate code duplication by using shared data arrays - Reduce code size by 45% while maintaining full functionality This resolves field misalignment issues and ensures sample data buttons generate consistent data based on the official field definitions. All 7 group fields are perfectly aligned with known fields JSON structure. * Make generate-known-fields script use dynamic version from package.json - Load version dynamically from plugins/main/package.json instead of hardcoding 4.14.0 - Add helper functions wazuhUrl() and dashboardPluginsUrl() for clean URL generation - Update all template URLs to use dynamic versioning - Ensure script remains sustainable across different Wazuh versions - Maintain backward compatibility and same functionality This resolves the issue where the script was hardcoded to version 4.14.0, making it unsustainable as branches update to newer versions. * Apply code formatting to generate-known-fields script - Improve readability with proper line breaks for long URLs - Format function calls with consistent indentation - Apply prettier-style formatting for better maintainability * Complete dynamic versioning for all template URLs in generate-known-fields script * Update scripts/README.md Co-authored-by: Antonio <34042064+Desvelao@users.noreply.github.com> * refactor: pass indexType as parameter to existsIndices function instead of inferring the index type internally * Apply code formatting to existsIndices calls - Format multi-line function calls for better readability - Apply consistent indentation style * Remove fallback to ALERTS in getIndexTypeFromPattern and specify index types explicitly * Use index type to get known fields and remove GeneratedKnownFields duplication * Move known fields JSON to common directory to fix server-client import issue - Move all known fields JSON files from public/utils/known-fields to common/known-fields - Update known-fields-reader.js to read from common instead of public - Update known-fields-loader.js and tests to import from common directory - Update generate-known-fields.js script to output to common directory - Fix production mode issue where server cannot import from public folder * Fix nested field handling in generate-known-fields script - Add proper handling for nested type fields with properties - Create field entry for the nested type itself before recursing into properties - Add 'nested' type to mapIndexFieldType function - Update isAggregatable and shouldReadFromDocValues to handle nested type correctly - Regenerate alerts.json with 7 additional nested fields (614 -> 621 fields) This fixes the issue where nested fields like configurationManagerClientEnabledFeatures were not being properly processed - now both the nested field and its sub-properties are correctly included in the known fields output. * Apply code formatting to generate-known-fields script * Simplify template extraction by converting TypeScript templates to JSON - Create monitoring-template.json from monitoring-template.ts - Create statistics-template.json from statistics-template.ts - Remove TypeScript parsing logic from generate-known-fields.js - Simplify fetchTemplate() to only handle JSON - Update script configuration to use .json files instead of .ts - Update README.md to reflect changes and improvements This addresses code review feedback to simplify template extraction by using JSON files instead of parsing TypeScript, making the script simpler and more maintainable. * Apply code formatting to statistics-template.json and README.md * Update README to reflect completed inventory states support - Remove outdated TODO about adding inventory states support (already implemented) - Update template sources section with complete list of supported templates - Clarify that all inventory states are now supported (13 types) - Remove references to TypeScript parsing issues (resolved) * Apply code formatting to monitoring-template.ts (add trailing commas) * Move known fields back to public and fix server path resolution - Move known-fields JSON files back from common/ to public/utils/known-fields/ - Update known-fields-loader.js to import from ./known-fields/ (public) - Update known-fields-reader.js (server) to dynamically resolve path - Add fallback path resolution for production vs development environments - Remove common/known-fields/ directory to avoid duplication - Update generate-known-fields.js to output to public/utils/known-fields/ The server now uses fs.existsSync() to check development path first, then falls back to production path, ensuring it works in both environments without duplicating files or hardcoding paths. This resolves webpack build errors where imports from common/ were not resolved correctly, while still addressing the code review concern about server-client imports by using dynamic path resolution. * Apply code formatting to generate-known-fields.js output paths * Exclude monitoring from auto-generation and use manual fields - Remove monitoring from generate-known-fields.js script configuration - Add TODO explaining why monitoring is not auto-generated - Delete auto-generated monitoring.json file - Update known-fields-loader.js to import from monitoring-fields.ts - Update test to use FieldsMonitoring from monitoring-fields.ts - Update README to document monitoring exclusion with explanation The monitoring template only contains basic index-level fields, but the actual monitoring index includes additional agent fields (dateAdd, lastKeepAlive, etc.) that are inserted dynamically by Wazuh server. These are maintained manually in monitoring-fields.ts to ensure all required fields are available. * Convert statistics-template from TypeScript to JSON - Add index_patterns property to statistics-template.json for dynamic assignment - Update import in scheduler-handler.ts to use JSON format - Remove statistics-template.ts as it's now replaced by JSON This aligns with the script-generated templates approach and fixes the 'Cannot set properties of undefined' error in the cron scheduler by ensuring the index_patterns property exists before assignment. * Fix extractPatternType regex to correctly handle patterns with multiple hyphens The previous regex pattern.match(/wazuh-states-(.+?)-\*/) used lazy matching which stopped at the first hyphen, incorrectly extracting: - 'wazuh-states-fim-files-*' → 'fim' (wrong, should be 'fim-files') - 'wazuh-states-inventory-system-*' → 'inventory' (wrong, should be 'inventory-system') This caused getKnownFieldsForStatesPattern() to return null for most states patterns with hyphens in their names. Solution: Replace regex with explicit string extraction: 1. Check if pattern starts with 'wazuh-states-' and ends with '-*' 2. Extract the substring between these markers 3. Verify extracted type exists in KnownFieldsStatesGenerated This approach is more robust and won't break if pattern prefixes change in the future, as it validates against the actual known pattern types. Added test case to verify unknown pattern types are correctly rejected. * Update changelog entry to be more user-oriented Changed from technical 'Automated known fields generation system' to user-friendly 'Add known fields for Wazuh states index patterns creation' which better describes the feature benefit for end users. * Apply code formatting (prettier) to modified files - Format check-index.ts - Format all known-fields JSON files - Format monitoring-template.json - Format known-fields-reader.js * Fix tests to use correct index type keys for getKnownFieldsByIndexType The function expects keys with 'wazuh-index-type-' prefix, not the raw constant values. Updated tests to use the correct string literals instead of the WAZUH_INDEX_TYPE_* constants which have different values. * Remove deprecated getKnownFieldsForStatesPattern function and its related tests. This change streamlines the known fields loader by eliminating unused code, improving maintainability. * Unify known fields mapping and fix getKnownFieldsByIndexType Created KnownFieldsByIndexType as a single unified mapping that uses the actual WAZUH_INDEX_TYPE_* constant values ('alerts', 'monitoring', etc.) as keys instead of prefixed strings. Benefits: - Eliminates inconsistency between constant values and map keys - getKnownFieldsByIndexType() now works correctly with constants - Single source of truth for all index types (main + states) - Kept KnownFieldsStatesGenerated for backwards compatibility - Tests now use constants directly instead of hardcoded strings This fixes the issue where getKnownFieldsByIndexType() was always returning null because it was looking for 'wazuh-index-type-*' keys that didn't exist in the map. * Use constants as keys in KnownFieldsByIndexType object Instead of hardcoded strings, now using computed property names with WAZUH_INDEX_TYPE_* constants for better maintainability and type safety. Example: Before: 'alerts': alertsFields After: [WAZUH_INDEX_TYPE_ALERTS]: alertsFields This ensures the keys are always in sync with the constant definitions and makes refactoring safer. * Refactor known fields loader by removing deprecated states pattern functions This commit eliminates the `KnownFieldsStatesGenerated` mapping and related functions, streamlining the known fields loader for better maintainability. The `getKnownFieldsForPattern` and `extractPatternType` functions have been removed as they are no longer necessary. The `KnownFieldsByIndexType` now serves as the single source of truth for all index types, enhancing clarity and reducing redundancy in the codebase. * Remove deprecated monitoring-template.json file to streamline integration files. This change eliminates unused configuration, enhancing maintainability and reducing clutter in the project. * Refactor sample data generation for states inventory groups This commit unifies the group data structure by consolidating descriptions, names, and user groups into a single format. The deprecated known fields reader has been removed, streamlining the sample data generation process. This change enhances maintainability and ensures consistency in the generated data. * Enhance index pattern validation by introducing indexType parameter This commit updates the ensureIndexPatternIsCreated function to accept an indexType parameter, allowing for more precise validation of index patterns. Additionally, various data source components have been modified to pass the appropriate indexType constants, improving maintainability and clarity in the codebase. Unused imports and deprecated functions have been removed to streamline the implementation. * Update CHANGELOG.md to reflect enhanced visualizations field validations for Wazuh states index patterns and support * Refactor pattern data source to ensure fields are updated correctly. This change moves the field retrieval and update logic into a try block, improving error handling and maintainability of the code. * Update pattern data source tests to improve error handling and assertions. The test for pattern not found now uses async/await syntax for clarity, and the test for index pattern field retrieval has been modified to ensure it does not throw errors when fields retrieval fails, reflecting the updated behavior of the select() method. * Add states inventory index type and associated fields This commit introduces the WAZUH_INDEX_TYPE_STATES_INVENTORY constant and integrates it into various components, including the validation logic and known fields loader. A new JSON file for states inventory fields is created, consolidating relevant data for improved management and accessibility. Additionally, a script is added to generate combined inventory fields, enhancing the overall structure and maintainability of the codebase. * Refactor combined inventory fields generation for improved readability This commit enhances the clarity of the combined inventory fields generation by formatting the function call to `generateCombinedInventoryFields` across multiple lines. This change improves code readability and maintainability without altering the functionality. --------- Signed-off-by: Felipe Gonzalez <felipegonzalezmv@hotmail.com> Co-authored-by: Antonio <34042064+Desvelao@users.noreply.github.com> Co-authored-by: Ian Yenien Serrano <63758389+yenienserrano@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change implements a comprehensive automated system for generating known fields for all Wazuh index patterns to prevent errors in visualizations and data grids when underlying indices don't exist.
Key Features:
generate-known-fields.js) automatically fetches and processes official Wazuh templates from GitHub repositoriesknown-fields-loader.jsmodule dynamically imports generated field definitionsTemplates Implemented:
Technical Implementation:
Code Cleanup:
known-fields.jsfile to eliminate code duplicationknown-fields-loader.jsmoduleWhen wazuh cannot provide field information during index pattern creation, the system now falls back to these precise, template-specific known fields, ensuring all fields used in visualizations and tables are available with correct types and properties.
Issues Resolved
Evidence
Test
To test this PR:
Test all dashboards (Mainly, It Hygiene, FIM, Vulnerabilities)
yarn test:jestto verify all tests passnode scripts/generate-known-fields.jsto test the generation scriptCheck List
yarn test:jest