Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add BlackHoleStorage towards 100k+ evaluations #105

Merged
merged 14 commits into from
Apr 13, 2020
Merged

Add BlackHoleStorage towards 100k+ evaluations #105

merged 14 commits into from
Apr 13, 2020

Conversation

c-bata
Copy link
Owner

@c-bata c-bata commented Apr 12, 2020 •
edited
Loading

Example

package main

import (
	"log"
	"math"

	"github.com/c-bata/goptuna"
	"github.com/c-bata/goptuna/cmaes"
)

func objective(trial goptuna.Trial) (float64, error) {
	x1, _ := trial.SuggestFloat("x1", -10, 10)
	x2, _ := trial.SuggestFloat("x2", -10, 10)
	return math.Pow(x1-2, 2) + math.Pow(x2+5, 2), nil
}

func main() {
	relativeSampler := cmaes.NewSampler(
		cmaes.SamplerOptionNStartupTrials(5))
	study, err := goptuna.CreateStudy(
		"goptuna-example",
		goptuna.StudyOptionStorage(goptuna.NewBlackholeStorage(20)),
		goptuna.StudyOptionRelativeSampler(relativeSampler),
	)
	if err != nil {
		log.Fatal("failed to create study:", err)
	}

	if err = study.Optimize(objective, 200); err != nil {
		log.Fatal("failed to optimize:", err)
	}

	v, err := study.GetBestValue()
	if err != nil {
		log.Fatal("failed to get best value:", err)
	}
	params, err := study.GetBestParams()
	if err != nil {
		log.Fatal("failed to get best params:", err)
	}
	log.Printf("Best evaluation=%f (x1=%f, x2=%f)",
		v, params["x1"].(float64), params["x2"].(float64))
}
(venv) $ go run _examples/cmaes/main.go 
2020/04/13 02:39:31 [INFO] Trial finished: trialID=0 state=Complete evaluation=47.674293
2020/04/13 02:39:31 [INFO] Trial finished: trialID=1 state=Complete evaluation=16.564975
2020/04/13 02:39:31 [INFO] Trial finished: trialID=2 state=Complete evaluation=22.229764
2020/04/13 02:39:31 [INFO] Trial finished: trialID=3 state=Complete evaluation=132.160176
2020/04/13 02:39:31 [INFO] Trial finished: trialID=4 state=Complete evaluation=38.056051
2020/04/13 02:39:31 [INFO] Trial finished: trialID=5 state=Complete evaluation=46.057644
2020/04/13 02:39:31 [INFO] Trial finished: trialID=6 state=Complete evaluation=22.140480
2020/04/13 02:39:31 [INFO] Trial finished: trialID=7 state=Complete evaluation=62.531182
2020/04/13 02:39:31 [INFO] Trial finished: trialID=8 state=Complete evaluation=103.705104
2020/04/13 02:39:31 [INFO] Trial finished: trialID=9 state=Complete evaluation=27.084488
2020/04/13 02:39:31 [INFO] Trial finished: trialID=10 state=Complete evaluation=33.224279
2020/04/13 02:39:31 [INFO] Trial finished: trialID=11 state=Complete evaluation=32.438836
2020/04/13 02:39:31 [INFO] Trial finished: trialID=12 state=Complete evaluation=62.485697
2020/04/13 02:39:31 [INFO] Trial finished: trialID=13 state=Complete evaluation=10.835063
2020/04/13 02:39:31 [INFO] Trial finished: trialID=14 state=Complete evaluation=61.231569
2020/04/13 02:39:31 [INFO] Trial finished: trialID=15 state=Complete evaluation=18.656672
2020/04/13 02:39:31 [INFO] Trial finished: trialID=16 state=Complete evaluation=20.223998
2020/04/13 02:39:31 [INFO] Trial finished: trialID=17 state=Complete evaluation=14.679579
2020/04/13 02:39:31 [INFO] Trial finished: trialID=18 state=Complete evaluation=2.867698
2020/04/13 02:39:31 [INFO] Trial finished: trialID=19 state=Complete evaluation=22.586033
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=20 state=Complete evaluation=44.850342
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=21 state=Complete evaluation=12.376161
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=22 state=Complete evaluation=14.530727
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=23 state=Complete evaluation=0.252565
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=24 state=Complete evaluation=23.930059
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=25 state=Complete evaluation=18.707611
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=26 state=Complete evaluation=23.723537
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=27 state=Complete evaluation=60.388024
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=28 state=Complete evaluation=49.107782
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=29 state=Complete evaluation=20.578979
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=30 state=Complete evaluation=17.343054
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=31 state=Complete evaluation=15.230447
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=32 state=Complete evaluation=0.688558
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=33 state=Complete evaluation=29.037456
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=34 state=Complete evaluation=30.733836
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=35 state=Complete evaluation=3.108564
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=36 state=Complete evaluation=15.524595
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=37 state=Complete evaluation=54.670613
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=38 state=Complete evaluation=5.800003
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=39 state=Complete evaluation=8.017986
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=40 state=Complete evaluation=12.064258
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=41 state=Complete evaluation=20.230681
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=42 state=Complete evaluation=1.552530
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=43 state=Complete evaluation=12.454509
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=44 state=Complete evaluation=14.703087
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=45 state=Complete evaluation=44.842796
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=46 state=Complete evaluation=28.649740
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=47 state=Complete evaluation=7.217258
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=48 state=Complete evaluation=0.151439
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=49 state=Complete evaluation=2.177288
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=50 state=Complete evaluation=1.167271
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=51 state=Complete evaluation=5.765259
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=52 state=Complete evaluation=0.662816
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=53 state=Complete evaluation=7.699746
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=54 state=Complete evaluation=9.101934
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=55 state=Complete evaluation=5.414616
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=56 state=Complete evaluation=4.842222
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=57 state=Complete evaluation=0.617716
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=58 state=Complete evaluation=17.855878
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=59 state=Complete evaluation=1.285338
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=60 state=Complete evaluation=0.894557
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=61 state=Complete evaluation=1.662849
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=62 state=Complete evaluation=2.419421
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=63 state=Complete evaluation=7.075800
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=64 state=Complete evaluation=0.977344
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=65 state=Complete evaluation=2.240027
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=66 state=Complete evaluation=4.759263
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=67 state=Complete evaluation=0.153983
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=68 state=Complete evaluation=0.658265
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=69 state=Complete evaluation=1.020960
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=70 state=Complete evaluation=0.569522
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=71 state=Complete evaluation=0.539054
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=72 state=Complete evaluation=2.391731
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=73 state=Complete evaluation=1.262207
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=74 state=Complete evaluation=1.410689
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=75 state=Complete evaluation=0.490438
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=76 state=Complete evaluation=1.446373
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=77 state=Complete evaluation=0.214436
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=78 state=Complete evaluation=0.707316
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=79 state=Complete evaluation=0.888024
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=80 state=Complete evaluation=1.500627
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=81 state=Complete evaluation=0.128145
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=82 state=Complete evaluation=0.170781
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=83 state=Complete evaluation=0.141563
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=84 state=Complete evaluation=0.153672
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=85 state=Complete evaluation=0.532152
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=86 state=Complete evaluation=1.286757
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=87 state=Complete evaluation=0.359358
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=88 state=Complete evaluation=0.152523
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=89 state=Complete evaluation=0.296147
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=90 state=Complete evaluation=0.261162
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=91 state=Complete evaluation=0.301497
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=92 state=Complete evaluation=0.064521
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=93 state=Complete evaluation=0.691118
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=94 state=Complete evaluation=0.070931
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=95 state=Complete evaluation=0.301691
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=96 state=Complete evaluation=0.078933
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=97 state=Complete evaluation=0.563108
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=98 state=Complete evaluation=0.045807
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=99 state=Complete evaluation=0.327774
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=100 state=Complete evaluation=0.009999
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=101 state=Complete evaluation=0.148794
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=102 state=Complete evaluation=0.018099
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=103 state=Complete evaluation=0.136851
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=104 state=Complete evaluation=0.042859
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=105 state=Complete evaluation=0.282536
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=106 state=Complete evaluation=0.125910
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=107 state=Complete evaluation=0.003086
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=108 state=Complete evaluation=0.124410
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=109 state=Complete evaluation=0.072585
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=110 state=Complete evaluation=0.000829
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=111 state=Complete evaluation=0.008112
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=112 state=Complete evaluation=0.058168
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=113 state=Complete evaluation=0.005940
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=114 state=Complete evaluation=0.030128
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=115 state=Complete evaluation=0.029122
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=116 state=Complete evaluation=0.125034
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=117 state=Complete evaluation=0.034411
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=118 state=Complete evaluation=0.077104
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=119 state=Complete evaluation=0.013749
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=120 state=Complete evaluation=0.000399
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=121 state=Complete evaluation=0.002622
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=122 state=Complete evaluation=0.003346
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=123 state=Complete evaluation=0.021465
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=124 state=Complete evaluation=0.016046
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=125 state=Complete evaluation=0.001959
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=126 state=Complete evaluation=0.022310
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=127 state=Complete evaluation=0.008421
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=128 state=Complete evaluation=0.000264
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=129 state=Complete evaluation=0.001306
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=130 state=Complete evaluation=0.007589
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=131 state=Complete evaluation=0.010711
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=132 state=Complete evaluation=0.000137
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=133 state=Complete evaluation=0.013548
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=134 state=Complete evaluation=0.007672
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=135 state=Complete evaluation=0.003877
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=136 state=Complete evaluation=0.007667
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=137 state=Complete evaluation=0.001077
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=138 state=Complete evaluation=0.000195
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=139 state=Complete evaluation=0.004136
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=140 state=Complete evaluation=0.008115
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=141 state=Complete evaluation=0.003044
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=142 state=Complete evaluation=0.006253
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=143 state=Complete evaluation=0.002411
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=144 state=Complete evaluation=0.001576
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=145 state=Complete evaluation=0.001547
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=146 state=Complete evaluation=0.002558
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=147 state=Complete evaluation=0.000411
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=148 state=Complete evaluation=0.000084
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=149 state=Complete evaluation=0.001228
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=150 state=Complete evaluation=0.000052
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=151 state=Complete evaluation=0.002752
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=152 state=Complete evaluation=0.003859
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=153 state=Complete evaluation=0.000480
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=154 state=Complete evaluation=0.001038
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=155 state=Complete evaluation=0.000385
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=156 state=Complete evaluation=0.001061
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=157 state=Complete evaluation=0.000026
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=158 state=Complete evaluation=0.000357
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=159 state=Complete evaluation=0.000382
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=160 state=Complete evaluation=0.000301
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=161 state=Complete evaluation=0.001494
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=162 state=Complete evaluation=0.001063
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=163 state=Complete evaluation=0.000044
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=164 state=Complete evaluation=0.000445
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=165 state=Complete evaluation=0.000032
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=166 state=Complete evaluation=0.000349
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=167 state=Complete evaluation=0.000036
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=168 state=Complete evaluation=0.000382
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=169 state=Complete evaluation=0.000045
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=170 state=Complete evaluation=0.000028
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=171 state=Complete evaluation=0.000061
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=172 state=Complete evaluation=0.000109
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=173 state=Complete evaluation=0.000014
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=174 state=Complete evaluation=0.000067
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=175 state=Complete evaluation=0.000191
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=176 state=Complete evaluation=0.000165
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=177 state=Complete evaluation=0.000002
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=178 state=Complete evaluation=0.000125
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=179 state=Complete evaluation=0.000056
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=180 state=Complete evaluation=0.000019
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=181 state=Complete evaluation=0.000028
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=182 state=Complete evaluation=0.000052
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=183 state=Complete evaluation=0.000025
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=184 state=Complete evaluation=0.000046
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=185 state=Complete evaluation=0.000006
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=186 state=Complete evaluation=0.000011
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=187 state=Complete evaluation=0.000075
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=188 state=Complete evaluation=0.000009
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=189 state=Complete evaluation=0.000026
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=190 state=Complete evaluation=0.000059
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=191 state=Complete evaluation=0.000028
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=192 state=Complete evaluation=0.000004
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=193 state=Complete evaluation=0.000015
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=194 state=Complete evaluation=0.000057
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=195 state=Complete evaluation=0.000011
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=196 state=Complete evaluation=0.000016
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=197 state=Complete evaluation=0.000012
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=198 state=Complete evaluation=0.000019
2020/04/13 02:39:31 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:39:31 [INFO] Trial finished: trialID=199 state=Complete evaluation=0.000002
2020/04/13 02:39:31 Best evaluation=0.000002 (x1=1.998966, x2=-5.000825)

Time: 10000 trials

$ time go run _examples/cmaes/main.go 
...
2020/04/13 02:44:33 [INFO] Trial finished: trialID=9999 state=Complete evaluation=NaN
2020/04/13 02:44:33 Best evaluation=0.000000 (x1=2.000000, x2=-5.000000)

real    0m2.370s
user    0m2.287s
sys     0m0.496s

Time: 100000 trials

 $ time go run _examples/cmaes/main.go
...
2020/04/13 02:41:47 [INFO] Trial finished: trialID=99998 state=Complete evaluation=NaN
2020/04/13 02:41:47 [WARN] IntersectionSearchSpace: GetTrials return some trials are already deleted:
2020/04/13 02:41:47 [INFO] Trial finished: trialID=99999 state=Complete evaluation=NaN
2020/04/13 02:41:47 Best evaluation=0.000000 (x1=2.000000, x2=-5.000000)

real    0m20.989s
user    0m20.829s
sys     0m1.421s

v.s. InMemoryStorage

storage InMemoryStorage BlackHoleStorage
10000 trials 1m1.88s 0m2.370s
100000 trials 96m26.527s 0m20.989s

@c-bata c-bata changed the title Add BlackHoleStorage for over 10k+ trials Add BlackHoleStorage towards 10k+ trials Apr 12, 2020
@c-bata c-bata marked this pull request as ready for review April 12, 2020 16:59
@github-actions
Copy link

github-actions bot commented Apr 12, 2020 •
edited
Loading

Benchmark result of rosenbrock function

plot curve image

  • Report ID: 709f41d11dcc604aa85c8b676009df2a5b9c4e56615ebbedff77d34568e88c2f
  • Kurobako Version: 0.1.4
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (CMA-ES) 1 1
Goptuna (TPE) 0 1
Random 0 0

Individual Results

(1) Problem: Rosenbrock Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (CMA-ES) (study) 0.054079 +- 0.083976 15606.304 +- 20497.028 0.070 +- 0.004
1 Goptuna (TPE) (study) 0.251651 +- 0.223612 423683.357 +- 351636.364 0.321 +- 0.019
2 Random (study) 1.512404 +- 1.571337 69830.718 +- 56547.971 0.000 +- 0.000

Solvers

ID: 55d05ec1515f9127963f4269fdf631907f78c963bff02793ff8c76b987bc277d

recipe:

{
  "command": {
    "path": "./cma_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 79277b8469b8810702825f63c5b450bba72028be027598049d9b7a5d94ccd4ce

recipe:

{
  "command": {
    "path": "./tpe_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 29b0b413e0e229de860c855bd7258ff2335c36821b0877f4c28fe96fc82afb3c

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.4"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 52cccf44ea75f7b3ffac45f5590f4957c127439c3a1685179124a94a06e16b64

recipe:

{
  "command": {
    "path": "./rosenbrock_problem",
    "args": []
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 49f9ee312dfcca494dca8fb37a2b167ecaa24509a795ae24efb7f1c695ac6da9

ID: 9393401716052aeaafc25616db33049d94c0cd2884bc6becab4e0dca520536f9

ID: 0a0b6c449c4456a2f1e8d0e919c0c8e6ccaf2d274ec43075abb065facdd30861

@github-actions
Copy link

github-actions bot commented Apr 12, 2020 •
edited
Loading

Benchmark result of himmelblau function

plot curve image

  • Report ID: bdfe7ce69fc21417d0eaf565aeeab319f180c92be85e719b419cfbb01860331c
  • Kurobako Version: 0.1.4
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (CMA-ES) 2 1
Goptuna (TPE) 0 0
Random 0 0

Individual Results

(1) Problem: Himmelblau Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (CMA-ES) (study) 0.000001 +- 0.000003 1273.126 +- 643.948 0.066 +- 0.006
2 Random (study) 0.445471 +- 0.399927 839.410 +- 365.782 0.000 +- 0.000
2 Goptuna (TPE) (study) 0.121484 +- 0.138786 871.734 +- 357.615 0.341 +- 0.012

Solvers

ID: 55d05ec1515f9127963f4269fdf631907f78c963bff02793ff8c76b987bc277d

recipe:

{
  "command": {
    "path": "./cma_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 79277b8469b8810702825f63c5b450bba72028be027598049d9b7a5d94ccd4ce

recipe:

{
  "command": {
    "path": "./tpe_solver",
    "args": []
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 29b0b413e0e229de860c855bd7258ff2335c36821b0877f4c28fe96fc82afb3c

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.4"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: fa9f14b4a97956cc4d0dd6769e8548cd5a12ae28520361741d847bd67d1cc511

recipe:

{
  "command": {
    "path": "./himmelblau_problem",
    "args": []
  }
}

specification:

{
  "name": "Himmelblau Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Himmelblau",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 8fbd46ba92dfcba2c6fac15cb53a81d261f30a2a7993b48024a012dd774671b9

ID: d39fa90bc8984e8b3690e27b79ca195bce3500a8b5873d2908b98cb34d2cd554

ID: fdcb4ad678796c7c8110d1aba73c9e9a31816a93aef77b8e5c8658d67d206a53

@c-bata c-bata changed the title Add BlackHoleStorage towards 10k+ trials Add BlackHoleStorage towards 100k+ evaluations Apr 12, 2020
@c-bata
Copy link
Owner Author

c-bata commented Apr 12, 2020 •
edited
Loading

Memory usage of InMemoryStorage

ScreenShot 2020-04-13 5 03 24

@c-bata c-bata merged commit a9d1276 into master Apr 13, 2020
@c-bata c-bata deleted the blackhole-storage branch April 13, 2020 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@c-bata