Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring task cache manager for forecasting #982

Merged
merged 3 commits into from
Aug 17, 2023

Commits on Aug 16, 2023

  1. Refactoring Shared Functionality for Broader Task Compatibility

    This PR extracts shared components from `ADTaskCacheManager` into `TaskCacheManager` to support the requirements of the forecasting feature.
    
    **Renamings**:
    
    - **Method-level in `ADTaskCacheManager`**:
      - `addDeletedDetector` to `addDeletedConfig`
      - `addDeletedDetectorTask` to `addDeletedTask`
      - `hasDeletedDetectorTask` to `hasDeletedTask`
      - `pollDeletedDetector` to `pollDeletedConfig`
      - `pollDeletedDetectorTask` to `pollDeletedTask`
    
    - **Variable-level in `AnomalyDetectorSettings`**:
      - `CHECKPOINT_MAINTAIN_QUEUE_MAX_HEAP_PERCENT` to `AD_CHECKPOINT_MAINTAIN_QUEUE_MAX_HEAP_PERCENT`
      - `CHECKPOINT_READ_QUEUE_MAX_HEAP_PERCENT` to `AD_CHECKPOINT_READ_QUEUE_MAX_HEAP_PERCENT`
      - `CHECKPOINT_SAVING_FREQ` to `AD_CHECKPOINT_SAVING_FREQ`
      - `CHECKPOINT_TTL` to `AD_CHECKPOINT_TTL`
      - `CHECKPOINT_WRITE_QUEUE_MAX_HEAP_PERCENT` to `AD_CHECKPOINT_WRITE_QUEUE_MAX_HEAP_PERCENT`
      - `COLD_ENTITY_QUEUE_MAX_HEAP_PERCENT` to `AD_COLD_ENTITY_QUEUE_MAX_HEAP_PERCENT`
      - `DEDICATED_CACHE_SIZE` to `AD_DEDICATED_CACHE_SIZE`
      - `ENTITY_COLD_START_QUEUE_CONCURRENCY` to `AD_ENTITY_COLD_START_QUEUE_CONCURRENCY`
      - `ENTITY_COLD_START_QUEUE_MAX_HEAP_PERCENT` to `AD_ENTITY_COLD_START_QUEUE_MAX_HEAP_PERCENT`
      - `EXPECTED_COLD_ENTITY_EXECUTION_TIME_IN_MILLISECS` to `AD_EXPECTED_COLD_ENTITY_EXECUTION_TIME_IN_MILLISECS`
      - `FILTER_BY_BACKEND_ROLES` to `AD_FILTER_BY_BACKEND_ROLES`
      - `MAX_ENTITIES_PER_QUERY` to `AD_MAX_ENTITIES_PER_QUERY`
      - `MAX_MODEL_SIZE_PER_NODE` to `AD_MAX_MODEL_SIZE_PER_NODE`
      - `MAX_MULTI_ENTITY_ANOMALY_DETECTORS` to `AD_MAX_HC_ANOMALY_DETECTORS`
      - `MAX_RETRY_FOR_END_RUN_EXCEPTION` to `AD_MAX_RETRY_FOR_END_RUN_EXCEPTION`
      - `MAX_SINGLE_ENTITY_ANOMALY_DETECTORS` to `AD_MAX_SINGLE_ENTITY_ANOMALY_DETECTORS`
      - `MODEL_MAX_SIZE_PERCENTAGE` to `AD_MODEL_MAX_SIZE_PERCENTAGE`
      - `PAGE_SIZE` to `AD_PAGE_SIZE`
      - `RESULT_WRITE_QUEUE_MAX_HEAP_PERCENT` to `AD_RESULT_WRITE_QUEUE_MAX_HEAP_PERCENT`
    
    - **Class-level**:
      - `ADRealtimeTaskCache` renamed to `RealtimeTaskCache`
    
    - **Package-level**:
      - Shifted from `org.opensearch.ad.breaker` to `org.opensearch.timeseries.breaker`
    
    **Migrations**:
    
    - Variables transferred from `AnomalyDetectorSettings` to `TimeSeriesSettings`:
      - `BATCH_BOUNDING_BOX_CACHE_RATIO`
      - `CHECKPOINT_MAINTAIN_REQUEST_SIZE_IN_BYTES`
      - `CHECKPOINT_WRITE_QUEUE_SIZE_IN_BYTES`
      - `DEFAULT_AD_JOB_LOC_DURATION_SECONDS` (renamed to `DEFAULT_JOB_LOC_DURATION_SECONDS`)
      - `ENTITY_REQUEST_SIZE_IN_BYTES`
      - `HOURLY_MAINTENANCE`
      - `INTERVAL_RATIO_FOR_REQUESTS`
      - `LOW_SEGMENT_PRUNE_RATIO`
      - `MAINTENANCE_FREQ_CONSTANT`
      - `MAX_COLD_START_ROUNDS`
      - `MAX_QUEUED_TASKS_RATIO`
      - `MAX_TOTAL_RCF_SERIALIZATION_BUFFERS`
      - `MAX_CHECKPOINT_BYTES`
      - `MEDIUM_SEGMENT_PRUNE_RATIO`
      - `NUM_MIN_SAMPLES`
      - `NUM_SAMPLES_PER_TREE`
      - `NUM_TREES`
      - `QUEUE_MAINTENANCE`
      - `RESULT_WRITE_QUEUE_SIZE_IN_BYTES`
      - `SERIALIZATION_BUFFER_BYTES`
      - `THRESHOLD_MIN_PVALUE`
      - `TIME_DECAY`
    
    **Deletions**:
    - Obsolete settings and methods:
      - `DESIRED_MODEL_SIZE_PERCENTAGE` in `AnomalyDetectorSettings`
      - `getDesiredModelSize` in `MemoryTracker`
    
    **Modifications**:
    - `MemoryTracker` enums renamed for clear differentiation between real-time and historical memory usage, adding `REAL_TIME_FORECASTER` for a harmonized single-stream and HC analysis approach.
    
    **Tests**:
    - Changes validated with a successful Gradle build.
    
    Signed-off-by: Kaituo Li <kaituo@amazon.com>
    kaituo committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    13c90a8 View commit details
    Browse the repository at this point in the history
  2. improve comments

    Signed-off-by: Kaituo Li <kaituo@amazon.com>
    kaituo committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    7f5a077 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. improve comments

    Signed-off-by: Kaituo Li <kaituo@amazon.com>
    kaituo committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    3860c77 View commit details
    Browse the repository at this point in the history