Support IPOP-CMA-ES algorithm. #131

c-bata · 2020-07-24T10:56:48Z

Paper: Auger, A., Hansen, N.: A restart CMA evolution strategy with increasing population size. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1769–1776 (2005a)

following todos:

Add IPOP-CMA-ES option for Goptuna suggestion backend of Kubeflow/Katib.

github-actions · 2020-07-24T10:58:49Z

Benchmark result of rosenbrock function

Report ID: 1eab990175affd52d87ee05634745e799cde9a8ce53f88bf498bda1a6a6f5ff6
Kurobako Version: 0.1.10
Number of Solvers: 4
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Goptuna (CMA-ES)	1	1
Goptuna (IPOP-CMA-ES)	1	1
Optuna (CMA-ES)	1	1
Random	0	0

Individual Results

(1) Problem: Rosenbrock Function

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Optuna (CMA-ES) (study)	0.082186 +- 0.199208	71443.923 +- 92351.625	0.490 +- 0.003
1	Goptuna (IPOP-CMA-ES) (study)	0.074908 +- 0.174223	33606.966 +- 34794.595	0.072 +- 0.005
1	Goptuna (CMA-ES) (study)	0.155898 +- 0.167171	14201.355 +- 16105.113	0.070 +- 0.004
4	Random (study)	3.211511 +- 1.837685	214799.631 +- 200711.654	0.000 +- 0.000

Solvers

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rosenbrock_problem",
    "args": []
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 5215baebf9217aa9dd2b3904fde351f6eba751db45f36a9b9f84e616754878ab

problem: Rosenbrock Function
solver: Goptuna (CMA-ES)
budget: 300
repeats: 8
concurrency: 1

ID: 80416880cba9442d494385831fbd39545320e4b25d7a75235a2531cec0885c62

problem: Rosenbrock Function
solver: Goptuna (IPOP-CMA-ES)
budget: 300
repeats: 8
concurrency: 1

ID: 7bb2fc6862c228414cf27578a36dc5484f1a7c29974c808644b6f243fda9a521

problem: Rosenbrock Function
solver: Optuna (CMA-ES)
budget: 300
repeats: 8
concurrency: 1

ID: ff11e028559c0dd7fba670fdbaa978c691eed425893e295908fde1ab9f6a3501

problem: Rosenbrock Function
solver: Random
budget: 300
repeats: 8
concurrency: 1

github-actions · 2020-07-24T11:28:06Z

Benchmark result of sigopt/evalset/Ackley problem

Report ID: d79458ae37d42e08b8da02f1053ebdc3f794d26f7fb04174dc5dfa577150b910
Kurobako Version: 0.1.10
Number of Solvers: 5
Number of Problems: 1
Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Overall Results

Solver	Borda	Firsts
Goptuna (CMA-ES)	3	1
Goptuna (IPOP-CMA-ES)	3	1
Goptuna (TPE)	1	0
Optuna (TPE)	1	0
Random	0	0

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=20)

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Goptuna (IPOP-CMA-ES) (study)	14.186801 +- 1.479997	1672.709 +- 67.316	0.070 +- 0.021
1	Goptuna (CMA-ES) (study)	15.066830 +- 0.969745	1689.507 +- 52.877	0.072 +- 0.018
3	Goptuna (TPE) (study)	16.775807 +- 0.609982	1804.515 +- 30.542	0.726 +- 0.150
3	Optuna (TPE) (study)	17.436996 +- 0.357783	1837.529 +- 31.856	13.748 +- 3.019
5	Random (study)	19.119660 +- 0.331625	1925.599 +- 16.654	0.000 +- 0.000

Solvers

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 5391658cb10db240ae15be38b5b37e83f4463dd3642a11bb1fbb5600d1c5f141

recipe:

{
  "sigopt": {
    "name": "ACKLEY",
    "dim": 20
  }
}

specification:

{
  "name": "sigopt/evalset/Ackley(dim=20)",
  "attrs": {
    "github": "https://github.com/sigopt/evalset",
    "paper": "Dewancker, Ian, et al. \"A strategy for ranking optimization methods using multiple criteria.\" Workshop on Automatic Machine Learning. 2016.",
    "version": "kurobako_problems=0.1.7"
  },
  "params_domain": [
    {
      "name": "p0",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p1",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p2",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p3",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p4",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p5",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p6",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p7",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p8",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p9",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p10",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p11",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p12",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p13",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p14",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p15",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p16",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p17",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p18",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p19",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Objective Value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: aa14eca0f146f6ad829722278a3475cd02f819ee4c034348fc99f1e7a34c9837

problem: sigopt/evalset/Ackley(dim=20)
solver: Goptuna (CMA-ES)
budget: 100
repeats: 10
concurrency: 1

ID: 0728013a30e19e63398ee33370df8972ab6ae1b294611bb78ea8f516a5e656b8

problem: sigopt/evalset/Ackley(dim=20)
solver: Goptuna (IPOP-CMA-ES)
budget: 100
repeats: 10
concurrency: 1

ID: 96f9b547442760d0b438d64de21efd5ec5c90c5c925b61061a32e60515893889

problem: sigopt/evalset/Ackley(dim=20)
solver: Goptuna (TPE)
budget: 100
repeats: 10
concurrency: 1

ID: 3876de7d62fb2f0bf37ba3f665c7ab0dd992b210ace85f4e7bf6c797ba267a9e

problem: sigopt/evalset/Ackley(dim=20)
solver: Optuna (TPE)
budget: 100
repeats: 10
concurrency: 1

ID: 76a4de94aaa57448def92b9151864f44c4f924fe1cbedbcb47160388b3a5561d

problem: sigopt/evalset/Ackley(dim=20)
solver: Random
budget: 100
repeats: 10
concurrency: 1

c-bata added 2 commits July 24, 2020 17:24

Add CMA-ES optimizer option for PopulationSize

e9ffde2

Implement ShouldStop() method and IPOP-CMA-ES.

3805f0c

This comment has been minimized.

Sign in to view

c-bata added 2 commits July 24, 2020 20:13

Add SamplerOptionIPop

2efa539

Run IPOP-CMA-ES benchmark

39588d7

c-bata mentioned this pull request Jul 24, 2020

Gaussian process surrogate models for the CMA-ES #91

Open

c-bata added 2 commits July 24, 2020 20:38

Add IPOP-CMA-ES on supported algorithm

162b52d

Add test case for CMA-ES termination

77654ff

c-bata marked this pull request as ready for review July 24, 2020 12:13

c-bata merged commit a4abd63 into master Jul 24, 2020

c-bata deleted the ipop-cma-es branch July 24, 2020 12:15

c-bata mentioned this pull request Jul 25, 2020

Randomize initial mean vector at restart #135

Merged

c-bata changed the title ~~Support IPOP-CMA-ES sampler.~~ Support IPOP-CMA-ES algorithm. Jul 25, 2020

Support IPOP-CMA-ES algorithm. #131

Support IPOP-CMA-ES algorithm. #131

Conversation

c-bata commented Jul 24, 2020 • edited Loading

This comment has been minimized.

github-actions bot commented Jul 24, 2020 • edited Loading

Benchmark result of rosenbrock function

Table of Contents

Overall Results

Individual Results

(1) Problem: Rosenbrock Function

Solvers

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

Studies

ID: 5215baebf9217aa9dd2b3904fde351f6eba751db45f36a9b9f84e616754878ab

ID: 80416880cba9442d494385831fbd39545320e4b25d7a75235a2531cec0885c62

ID: 7bb2fc6862c228414cf27578a36dc5484f1a7c29974c808644b6f243fda9a521

ID: ff11e028559c0dd7fba670fdbaa978c691eed425893e295908fde1ab9f6a3501

github-actions bot commented Jul 24, 2020 • edited Loading

Benchmark result of sigopt/evalset/Ackley problem

Table of Contents

Overall Results

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=20)

Solvers

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

Problems

ID: 5391658cb10db240ae15be38b5b37e83f4463dd3642a11bb1fbb5600d1c5f141

Studies

ID: aa14eca0f146f6ad829722278a3475cd02f819ee4c034348fc99f1e7a34c9837

ID: 0728013a30e19e63398ee33370df8972ab6ae1b294611bb78ea8f516a5e656b8

ID: 96f9b547442760d0b438d64de21efd5ec5c90c5c925b61061a32e60515893889

ID: 3876de7d62fb2f0bf37ba3f665c7ab0dd992b210ace85f4e7bf6c797ba267a9e

ID: 76a4de94aaa57448def92b9151864f44c4f924fe1cbedbcb47160388b3a5561d

c-bata commented Jul 24, 2020 •

edited

Loading

github-actions bot commented Jul 24, 2020 •

edited

Loading

github-actions bot commented Jul 24, 2020 •

edited

Loading