Pr branch - Push 2#12
Open
rakshiths-vunet wants to merge 134 commits into
Open
Conversation
A Draft Feature
…et support files.. Refactored main.go
…binary on node enable/disable actions
- Updated kafka_summary_processor.go to improve JSON marshaling for node CPU and memory metrics, ensuring proper error handling and logging. - Introduced node_json_processor.go to handle node resource metrics summarization, including fetching distinct node names and aggregating CPU and memory statistics. - Modified pod_resource_processor.go to include node names in pod metrics, enhancing the granularity of resource usage data. - Added new functions to fetch pipeline and Traefik pod metrics with node information, allowing for better monitoring of resource allocation. - Updated kafka-metrics.js to support multiple casing formats for test run IDs and durations, improving compatibility with API responses.
Add support for filtering K6 metrics by specific test runs using test_id. Updated data structures to include TestID fields, modified database queries to support test_id filtering with extended time windows, added new API endpoints for test ID retrieval and dropdown data, and enhanced UI with test run selection dropdown and updated tables to display test_id columns.
… reports - Introduce new functions for node-pod resource tables with rowspan - Add ClickHouse ingestion summary table parsing EPS metrics - Refactor pod and node metrics formatting to use JSON data directly - Update pipeline pod detection and formatting based on pipeline name - Automatically add '%' to Min/Avg/Max headers in tables fix(k6): improve query filtering for test_id in K6 result functions - Dynamically build WHERE clauses for test_id and time filters - Ensure historical data is shown when test_id is specified chore(config): remove unused Apache input topic from topics_tables.yaml refactor(node-processor): remove node name trimming in CPU and memory queries - Simplify node name handling in ClickHouse queries for better performance
…nfra and UI views - Add generate_combined_report.py to create HTML reports combining vudatasim and k6 data with interactive toggle - Update generate_html_reports.py and k6.py for improved report generation, including pod restarts and compressed CSS - Enhance pod metrics processing with restarts tracking and topic config loading from YAML - Add reports API handler and frontend reports viewer with PDF download capability - Update Kafka pod memory manager with test run filtering and time range support - Modify database schema to include pod_restarts column and update related processors - Improve Kafka metrics with input/output topic detection and error handling
- Update CompleteTimedOutTestRuns to check binary statuses and complete all running tests if no binaries are active - Add reports button to main navigation in index.html - Modify monitoring.html to comment out unused metrics and adjust filters - Overhaul reports.html with left-side drawer, toggle tab, and HTML download functionality - Comment out ClickHouse and monitoring event listeners in main.js BREAKING CHANGE: CompleteTimedOutTestRuns now requires a BinaryControl parameter
…ks to critical sections of the code. This ensures thread-safe operations when enabling/disabling sources and calculating EPS. Also, updated static/reports.html to correctly categorize and display violations for Kafka, ClickHouse, and Pipeline components separately.
…ion handling Refactor OAuth config into a reusable function, add comprehensive debug logging throughout auth handlers, improve session store initialization with proper options, and enhance middleware with access logging. This improves debugging capabilities and error handling without altering core functionality.
- Implement modal for creating JIRA tickets from violation reports - Add prefilled form fields based on violation type and component - Include functionality to open modal, generate form, and simulate ticket creation - Update logout redirect to /login page - Enhance logout button styling with icon and hover effects - Simplify user info display to show only name - Comment out global realtimeManager assignment in monitoring script - Change page title from "Load Test Dashboard" to "Reports
Previously, when no binaries were running, all running test runs were completed immediately. This change adds a condition to only complete test runs that have been running for at least 30 seconds, preventing premature completion of short-lived tests.
…proved maintainability - Extracted hardcoded strings into named constants for error messages, time formats, and commands - Added nil checks for BinaryControl instance in all public methods to prevent panics - Minor formatting adjustments to struct field alignments for consistency This refactoring enhances code readability, reduces duplication, and improves error handling without altering functionality.
- Add new HTML report for 'api k6 chumma' test with performance metrics - Implement request timeouts (15s) and abort controllers for API calls - Add resource cleanup methods to prevent memory leaks in managers - Improve error handling and initialization with graceful degradation - Stop auto-updates when filtering by test run to avoid conflicts - Add conditional auth bypass via DISABLE_AUTH environment variable - Adjust test completion checker timing for better reliability
Add new POST /api/start-vudatasim endpoint that performs complete vuDataSim startup sequence including test run creation, EPS split, distribution, Kafka recreation, ClickHouse truncate, and binary start.
Add new database columns for min, max, and avg bytes in/out per second for input and output topics. Implement KafkaBytesProcessor to fetch and aggregate bytes metrics from ClickHouse, and integrate into the summary processor for comprehensive test run reporting.
- Remove numerous unused columns from test_runs table (e.g., individual node CPU/memory metrics, pod resource checks) - Add new columns for process rate summary, ingestion summary, and pipeline info - Update queries in test_runs.go to handle time parsing and add GetTestRunO11ySources function - Refactor pod_resource_processor.go to use memory_working_set_bytes and improve restart calculations - Modify kafka_summary_processor.go to aggregate metrics per second and auto-execute combined report script - Add argparse support in report generation scripts for test-id filtering - Update configs for new hosts, node enablement, and Kubernetes namespace - Comment out pod deletion scheduling in UI BREAKING CHANGE: removed multiple columns from test_runs table, affecting data retrieval and storage for test runs. Update any dependent code accordingly.
- Added new topic configuration for Apache logs in topics_tables.yaml. - Updated database migrations to include new columns for lag metrics and topic metrics JSON. - Modified GetTestRun and GetAllTestRuns functions to retrieve and store topic metrics JSON. - Implemented per-source lag processing in consumer lag processor, including detailed logging. - Introduced detailed topic metrics computation in Kafka summary processor. - Updated TestRun model to include lag_per_source and topic_metrics_json fields. - Enhanced logging for better traceability during processing.
…simulation tools - Introduce k6 scripts for login testing and dashboard load simulation - Add Playwright-based automated dashboard performance testing with concurrent user simulation - Implement user creation utilities for Keycloak and cookie management - Update EPS configurations and enable/disable monitoring modules for optimized load testing - Enhance log analytics with updated scripts and configurations BREAKING CHANGE: Configuration changes may require updates to existing deployment scripts and monitoring setups.
…ation - Add SpringBoot as new observability source with complete configuration files - Include detailed project documentation covering architecture, usage, and reporting - Refactor node resource processors to use JSON-based metrics collection - Update configuration files for improved paths and settings - Remove deprecated process and node-specific processor files - Enhance UI with better node management and real-time monitoring - Update Kafka and ClickHouse handlers for improved error handling and configuration loading BREAKING CHANGE: Removed src/handlers/process.go and node-specific processors, update any direct imports accordingly.
- Added new API endpoints for starting, stopping, and checking the status of traces simulations. - Created a new handler for managing traces simulation requests. - Introduced a TracesSimulator struct to manage the simulation process, including starting and stopping the simulation. - Updated main.go to include new routes for traces simulation. - Developed a new traces.js file for handling frontend interactions with the traces simulation UI. - Enhanced the HTML interface to include a dedicated section for traces simulation controls. - Added configuration files and main.go for the traces simulator application. - Updated ClickHouse and Kafka configurations to support traces data handling. - Implemented error handling and logging for better traceability during simulation operations.
…BI YonoAuditTransLogs generator (Sirish), and update K6 scripts and Apache configurations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.