[feat] Add an object that realizes the perf over time viz (#331)

* [feat] Add an object that realizes the perf over time viz

* [fix] Modify TODOs and add comments to avoid complications

* [refactor] [feat] Format visualizer API and integrate this feature into BaseTask

* [refactor] Separate a shared raise error process as a function

* [refactor] Gather params in Dataclass to look smarter

* [refactor] Merge extraction from history to the result manager

Since this feature was added in a previous PR, we now rely on this
feature to extract the history.
To handle the order by the start time issue, I added the sort by endtime
feature.

* [feat] Merge the viz in the latest version

* [fix] Fix nan --> worst val so that we can always handle by number

* [fix] Fix mypy issues

* [test] Add test for get_start_time

* [test] Add test for order by end time

* [test] Add tests for ensemble results

* [test] Add tests for merging ensemble results and run history

* [test] Add the tests in the case of ensemble_results is None

* [fix] Alternate datetime to timestamp in tests to pass universally

Since the mapping of timestamp to datetime variates on machine,
the tests failed in the previous version.
In this version, we changed the datetime in the tests to the fixed
timestamp so that the tests will pass universally.

* [fix] Fix status_msg --> status_type because it does not need to be str

* [fix] Change the name for the homogeniety

* [fix] Fix based on the file name change

* [test] Add tests for set_plot_args

* [test] Add tests for plot_perf_over_time in BaseTask

* [refactor] Replace redundant lines by pytest parametrization

* [test] Add tests for _get_perf_and_time

* [fix] Remove viz attribute based on Ravin's comment

* [fix] Fix doc-string based on Ravin's comments

* [refactor] Hide color label settings extraction in dataclass

Since this process makes the method in BaseTask redundant and this was
pointed out by Ravin, I made this process a method of dataclass so that
we can easily fetch this information.
Note that since the color and label information always depend on the
optimization results, we always need to pass metric results to ensure
we only get related keys.

* [test] Add tests for color label dicts extraction

* [test] Add tests for checking if plt.show is called or not

* [refactor] Address Ravin's comments and add TODO for the refactoring

* [refactor] Change KeyError in EnsembleResults to empty

Since it is not convenient to not be able to instantiate EnsembleResults
in the case when we do not have any histories,
I changed the functionality so that we can still instantiate even when
the results are empty.
In this case, we have empty arrays and it also matches the developers
intuition.

* [refactor] Prohibit external updates to make objects more robust

* [fix] Remove a member variable _opt_scores since it is confusing

Since opt_scores are taken from cost in run_history and metric_dict
takes from additional_info, it was confusing for me where I should
refer to what. By removing this, we can always refer to additional_info
when fetching information and metrics are always available as a raw
value. Although I changed a lot, the functionality did not change and
it is easier to add any other functionalities now.

* [example] Add an example how to plot performance over time

* [fix] Fix unexpected train loss when using cross validation

* [fix] Remove __main__ from example based on the Ravin's comment

* [fix] Move results_xxx to utils from API

* [enhance] Change example for the plot over time to save fig

Since the plt.show() does not work on some environments,
I changed the example so that everyone can run at least this example.

Author: nabenabe0928

autoPyTorch/api/base_task.py

@@ -21,6 +21,8 @@
 
 import joblib
 
+import matplotlib.pyplot as plt
+
 import numpy as np
 
 import pandas as pd
@@ -29,7 +31,7 @@
 from smac.stats.stats import Stats
 from smac.tae import StatusType
 
-from autoPyTorch.api.results_manager import ResultsManager, SearchResults
+from autoPyTorch import metrics
 from autoPyTorch.automl_common.common.utils.backend import Backend, create
 from autoPyTorch.constants import (
     REGRESSION_TASKS,
@@ -58,6 +60,8 @@
 )
 from autoPyTorch.utils.parallel import preload_modules
 from autoPyTorch.utils.pipeline import get_configuration_space, get_dataset_requirements
+from autoPyTorch.utils.results_manager import MetricResults, ResultsManager, SearchResults
+from autoPyTorch.utils.results_visualizer import ColorLabelSettings, PlotSettingParams, ResultsVisualizer
 from autoPyTorch.utils.single_thread_client import SingleThreadedClient
 from autoPyTorch.utils.stopwatch import StopWatch
 
@@ -1479,3 +1483,56 @@ def sprint_statistics(self) -> str:
             scoring_functions=self._scoring_functions,
             metric=self._metric
         )
+
+    def plot_perf_over_time(
+        self,
+        metric_name: str,
+        ax: Optional[plt.Axes] = None,
+        plot_setting_params: PlotSettingParams = PlotSettingParams(),
+        color_label_settings: ColorLabelSettings = ColorLabelSettings(),
+        *args: Any,
+        **kwargs: Any
+    ) -> None:
+        """
+        Visualize the performance over time using matplotlib.
+        The plot related arguments are based on matplotlib.
+        Please refer to the matplotlib documentation for more details.
+
+        Args:
+            metric_name (str):
+                The name of metric to visualize.
+                The names are available in
+                    * autoPyTorch.metrics.CLASSIFICATION_METRICS
+                    * autoPyTorch.metrics.REGRESSION_METRICS
+            ax (Optional[plt.Axes]):
+                axis to plot (subplots of matplotlib).
+                If None, it will be created automatically.
+            plot_setting_params (PlotSettingParams):
+                Parameters for the plot.
+            color_label_settings (ColorLabelSettings):
+                The settings of a pair of color and label for each plot.
+            args, kwargs (Any):
+                Arguments for the ax.plot.
+        """
+
+        if not hasattr(metrics, metric_name):
+            raise ValueError(
+                f'metric_name must be in {list(metrics.CLASSIFICATION_METRICS.keys())} '
+                f'or {list(metrics.REGRESSION_METRICS.keys())}, but got {metric_name}'
+            )
+        if len(self.ensemble_performance_history) == 0:
+            raise RuntimeError('Visualization is available only after ensembles are evaluated.')
+
+        results = MetricResults(
+            metric=getattr(metrics, metric_name),
+            run_history=self.run_history,
+            ensemble_performance_history=self.ensemble_performance_history
+        )
+
+        colors, labels = color_label_settings.extract_dicts(results)
+
+        ResultsVisualizer().plot_perf_over_time(  # type: ignore
+            results=results, plot_setting_params=plot_setting_params,
+            colors=colors, labels=labels, ax=ax,
+            *args, **kwargs
+        )