Kepler Model and Metrics Validation

[KaiyiLiu1234]

I thought it would be a good idea to create a discussion section to cement goals, concerns, and tasks for Kepler Metrics Validation (as discussed in the March 5, 2024 Community Meeting). At the moment @rootfs, I was wondering if it would be a good idea to start exploring the "Develop Validation Method" Task to try and generate a visual graph of the energy metrics received from BM vs VM and get a preliminary idea of what the current error is. So i would like to begin work on the "VM Node Power Calibration" Test Case. Specifically, I would like to first try and set up comparing Kepler on BM vs VM on a kind cluster using Kubevirt and basic scraper to compare the energy metrics, and then see from there. Note: use curve-fit models

[husky-parul]

I started doing research on this and can definitely use some help. Few things I would like to point out from the preliminary research.

1. To have more accurate calibration we need to ensure that we are using the same CPU architecture for the BM ,on which the models were trained and the VM were the model will be used for the estimation.
2. It is quite hard to procure VM with the same CPU architecture on the CI environment as well as locally.
3. So I started with a local environment on my fedora laptop (BM) and trying to enforce Kepler to use model to estimate power instead of reading from power source for Node Power. I encountered multiple issues with this technique. First, Kepler was always using the label as RAPL for source. This was fixed in a PR. Secondly, Kepler is using the source as None even when ACPI is available. I am still investigating this. Lastly, we need to modify the code, AFAIK, to enforce Kepler to use models. I have tried to use node model estimation as true in Kepler Config Map but that didn’t seem to do the trick. Maybe if a flag is introduced, which if set True, Kepler uses model for estimation even when it is running on BM might work.

@sustainable-computing-io/maintainer
Please share your views on this.

Thanks

[KaiyiLiu1234]

I would love to help out. Thanks a bunch for the research points you found. For my assigned experiment which was from the March 5 Community Meeting, I will be using two keplers which should bypass the issue on 3. That said, one of the test cases is baremetal node power validation which seeks to compare baremetal rapl node power with the model estimated node power directly, which sounds just like the work you are currently doing, so I agree a flag should be set up to allow Kepler to use models for estimation.

I think for 2, I believe we can use KVM hypervisor which can emulate x86_64 (which I think is the what we used to train a lot of the curve fit models?).

[husky-parul]

Hi @sustainable-computing-io/maintainer

I am proposing the we export the reports in json format. This is help in analytics on the report.

Json example

{
  "report": 
    {
      "build_info": {
        "metal": {
          "arch": "amd64",
          "branch": "validator",
          "instance": "kepler-metal:8888",
          "job": "metal",
          "os": "linux",
          "revision": "006370e4ae63a11f21bc2fd7543a73e915cc1fa0",
          "version": "metal"
        }
      },
      "vm": {
        "goarch": "amd64",
        "goos": "linux",
        "goversion": "go1.21.9 (Red Hat 1.21.9-2.el9_4)",
        "instance": "vm:9100",
        "job": "vm",
        "revision": "unknown",
        "tags": [
          "include_gcs",
          "include_oss",
          "containers_image_openpgp",
          "gssapi",
          "providerless",
          "netgo,osusergo",
          "libbpf",
          "linux",
          "gpu"
        ],
        "version": "latest"
      },
      "specs": {
        "host": {
          "cpu": "",
          "dram": ""
        },
        "vm": {
          "cpu": "",
          "dram": ""
        },
        "execution_time": "",
        "validation-result": [
          {
            "metric_name": "platform",
            "mode": "dynamic",
            "mse": "2121.4373702905764",
            "mape": "100.0"
          },
          {
            "metric_name": "package",
            "mode": "dynamic",
            "mse": "1.6873386411764675",
            "mape": "100.0"
          },
          {
            "metric_name": "core",
            "mode": "dynamic",
            "mse": "1.2433084870588234",
            "mape": "100.0"
          },
          {
            "metric_name": "dram",
            "mode": "dynamic",
            "mse": "0.0",
            "mape": "nan"
          },
          {
            "metric_name": "platform",
            "mode": "idle",
            "mse": "7.0908837847059285",
            "mape": "100.0"
          },
          {
            "metric_name": "package",
            "mode": "idle",
            "mse": "0.00012067941176470487",
            "mape": "100.0"
          },
          {
            "metric_name": "core",
            "mode": "idle",
            "mse": "4.000000000000011e-06",
            "mape": "100.0"
          },
          {
            "metric_name": "dram",
            "mode": "idle",
            "mse": "0.0",
            "mape": "nan"
          }
        ]
      }
    } 
}

@sthaha @dave-tucker