Closed
Description
The current workflow for memory limits is:
- User specifies a memory limit up front with no guidance as to what is sensible/realistic
- They start the analysis
- Analysis chugs away for a while reindexing the source index into the destination index
- C++ process is started and checks whether the memory limit is sufficient to do the analysis
- If it's not then the process exits, reporting how much memory was required in the error message
Two possible ways to solve it are:
- We duplicate that logic that calculates the memory requirement from the C++ code into the Java and UI code
- We add a mode of operation to the C++ process where you just supply the spec and instead of actually doing the analysis it just tells you:
i. What you'd have to set the memory limit to to do the analysis entirely in RAM
ii. What the minimum memory limit is that will enable the analysis to run at all (using disk)
Although option 2 is a lot of work, the memory calculations done by the C++ are now so complex that it is impractical to duplicate them. Therefore we should implement option 2. The work will be something along the lines of:
- Add a mode of operation to the C++ process where you just supply the spec and it returns the sizing information (i and ii above)
Introduce memory usage estimation mode in data_frame_analyzer ml-cpp#584 - Add an endpoint to the Java code that runs the C++ process in this mode and returns the output in the endpoint response
Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint #45188 - Call this endpoint early in the start data frame analytics endpoint sequence (before creating the persistent task) and fail early if the configured
model_memory_limit
is so low that the C++ process will fail immediately when run
Call the new _estimate_memory_usage API endpoint on df analytics _start #45536 - Call the endpoint as part of the configuration process in the UI so that the UI can present a sensible default
model_memory_limit
, advise on sensible upper and lower bounds and validate any value the user chooses themselves
[ML] Improve UX regarding df analytics model memory limit kibana#43740 - HLRC
HLRC for memory usage estimation API #45531 - docs
Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint #45188 (API reference), Add docs for HLRC for Estimate memory usage API #45538 (HLRC)