Add Das-Dennis weight initialization method #295

jonpsy · 2021-06-18T06:48:26Z

Continuing #293

Add uniform_init.hpp.
Use DefaultMOEAD = MOEAD<Uniform, Tchebycheff>.
Use BBSMOEAD = MOEAD<BayesianBootstrap, Tchebycheff>
Test and verify its working locally.
Document uniform_init.hpp.
Add documentation in optimizers.md.
HISTORY.md.

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

jonpsy · 2021-06-18T08:36:42Z

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

+    auto BinomialCoefficient =
+        [](size_t n, size_t k) -> double
+        {
+          return std::tgamma(n + 1) / (std::tgamma(k + 1) * std::tgamma(n - k + 1));


I was wondering why there's an integer overflow happening here; then it hit me. std::tgamma(300) is shooting off to infinity. This is a very naive approach to finding Binomial Coefficient, I guess I could utilize the fact that the denominator and numerator cancel out a lot of terms. Though it'll certainly be nice if armadillo had a support for this algorithm built-in cc/ @conradsnicta

jonpsy · 2021-06-18T09:31:54Z

The build failure seems unrelated to the PR. @zoq your insights would be helpful, thanks.

EDIT: It does look related but I don't get how. I haven't even anything out of the ordinary.

jonpsy · 2021-06-18T09:51:43Z

I'm getting errors on adagrad_test.cpp for the changes I did in moead class. I'm done with C++ 👍 .

zoq · 2021-06-18T12:28:56Z

I'm getting errors on adagrad_test.cpp for the changes I did in moead class. I'm done with cpp .

Huh, do you have a link to the Travis build, looks like it's down for me.

jonpsy · 2021-06-18T12:38:54Z

Fixed it, I was missing a brace. C++ is so fun

=> Use 300 instead of 150 population => DefaultMOEAD = MOEAD<Uniform, Tchebycheff>

zoq · 2021-06-18T15:24:48Z

include/ensmallen_bits/moead/moead.hpp

@@ -72,8 +74,8 @@ class MOEAD {
   * @param upperBound The upper bound on each variable of a member
   *    of the variable space.
   */
-  MOEAD(const size_t populationSize = 150,
-        const size_t maxGenerations = 300,
+  MOEAD(const size_t populationSize = 300,


Was the test failing with populationSize = 150?

This is because the binomial coefficient value (H) should equate to the size of the population. This algorithm is restrictive in that way. You can read Section IV (If I recall correctly) of the cited paper to know more. To ensure this, you can find I've added an if statement inside Generate method to catch a mismatch.

The main point is, there is only a certain number of points that can be generated uniformly given the number of partitions(gaps between points) and the number of objectives.

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

zoq · 2021-06-18T15:28:23Z

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

+        point.insert_rows(point.n_rows, RowType(1).fill(
+            delta * static_cast<ElemType>(beta)));


It's not super clean, but I think it works, at least this way we can avoid the std::vector conversion.

zoq · 2021-06-18T15:29:59Z

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

+    auto BinomialCoefficient =
+        [](size_t n, size_t k) -> size_t
+        {
+          size_t retval = 1;
+          // Since, C(n, k) = C(n, n - k).
+          if (k > n - k)
+            k = n - k;
+
+          // [n * (n - 1) * .... * (n - k + 1)] / [k * (k - 1) * .... * 1].
+          for (size_t i = 0; i < k; ++i)
+          {
+            retval *= (n - i);
+            retval /= (i + 1);
+          }
+
+          return retval;
+        };
+    return BinomialCoefficient(numObjectives + numPartitions - 1, numPartitions);


Note to myself, check that part in more depth.

Add this to your notes.

So, thinking about this... The lambda usage is nice, but should we maybe prefer to write this as a standalone function? Similar to the link post? Also, I think the code on the link post is 1-1 with what is written here. Do we need to worry about licensing?

The reason for making it lambda inside the function was to secure the scope, it won't be used elsewhere. Regarding the heavy correlation, GFG (Geeks for geeks) is kinda open-sourced, and is made to help people(like Stack Overflow). Besides, finding Binomial Coefficient in O(n) isn't a ground breaking research that we're using without citing :) .

According to https://www.geeksforgeeks.org/copyright-information/ this is fine, but let's add a comment which references the source for the code.

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

jonpsy · 2021-06-18T17:38:33Z

Could you re-trigger the CI please? The build was cancelled for some reason. Thanks.

zoq · 2021-06-18T17:40:57Z

Could you re-trigger the CI please? The build was cancelled for some reason. Thanks.

Done.

jonpsy · 2021-06-18T17:55:30Z

It got cancelled again...

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

…iform

jonpsy · 2021-06-18T18:38:54Z

Added using BBSMOEAD = MOEAD<BayesianBootstrap, Tchebycheff> along with documentation in optimizers.md.

jonpsy · 2021-06-19T12:26:32Z

@coatless Would you like to have a pass over this? Thanks.

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

zoq

Looks great to me.

HISTORY.md

coatless

See notes.

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

coatless · 2021-06-22T16:55:12Z

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp

+    auto BinomialCoefficient =
+        [](size_t n, size_t k) -> size_t
+        {
+          size_t retval = 1;
+          // Since, C(n, k) = C(n, n - k).
+          if (k > n - k)
+            k = n - k;
+
+          // [n * (n - 1) * .... * (n - k + 1)] / [k * (k - 1) * .... * 1].
+          for (size_t i = 0; i < k; ++i)
+          {
+            retval *= (n - i);
+            retval /= (i + 1);
+          }
+
+          return retval;
+        };
+    return BinomialCoefficient(numObjectives + numPartitions - 1, numPartitions);


So, thinking about this... The lambda usage is nice, but should we maybe prefer to write this as a standalone function? Similar to the link post? Also, I think the code on the link post is 1-1 with what is written here. Do we need to worry about licensing?

mlpack-bot

Second approval provided automatically after 24 hours. 👍

jonpsy · 2021-06-23T14:40:28Z

Great, now I have to wait another 2 days.

coatless · 2021-06-23T14:41:51Z

@jonpsy not at all! From the first approval it will automatically give a second after 24 hours. Once the second bot-approval is in, we can merge. :)

mlpack-bot bot added s: needs review s: unanswered s: unlabeled labels Jun 18, 2021

jonpsy commented Jun 18, 2021

View reviewed changes

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp Outdated Show resolved Hide resolved

jonpsy commented Jun 18, 2021

View reviewed changes

zoq added c: optimizers t: added feature and removed s: unanswered s: unlabeled labels Jun 18, 2021

jonpsy added 4 commits June 18, 2021 18:09

=> Added uniform init

60f9264

=> Use 300 instead of 150 population => DefaultMOEAD = MOEAD<Uniform, Tchebycheff>

efficient Binomial coefficient O(n)

96a1998

fix test params

d7be57a

Add the braces

275ded3

jonpsy force-pushed the uniform branch from cc5361c to 275ded3 Compare June 18, 2021 12:40

jonpsy added 7 commits June 18, 2021 18:11

add history

501d486

document in optimizers.md

cf1a9e7

RDP test

dbfcf7e

minor indent fix

7ebc527

oldschool

a8c664e

nit

d98f825

revert to normal

f9c5d7f

zoq reviewed Jun 18, 2021

View reviewed changes

jonpsy and others added 3 commits June 18, 2021 21:13

Update include/ensmallen_bits/moead/weight_init_policies/uniform_init…

3c0b467

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

Update include/ensmallen_bits/moead/weight_init_policies/uniform_init…

50f1245

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

Update include/ensmallen_bits/moead/weight_init_policies/uniform_init…

cb3dc29

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

zoq reviewed Jun 18, 2021

View reviewed changes

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp Outdated Show resolved Hide resolved

jonpsy and others added 3 commits June 18, 2021 23:52

Update include/ensmallen_bits/moead/weight_init_policies/uniform_init…

4f297c7

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

add BBSMOEAD

2399161

Merge branch 'uniform' of https://github.com/jonpsy/ensmallen into un…

3268605

…iform

coatless reviewed Jun 21, 2021

View reviewed changes

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp Show resolved Hide resolved

zoq reviewed Jun 21, 2021

View reviewed changes

include/ensmallen_bits/moead/weight_init_policies/uniform_init.hpp Outdated Show resolved Hide resolved

Update include/ensmallen_bits/moead/weight_init_policies/uniform_init…

bb893f6

….hpp Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

zoq approved these changes Jun 21, 2021

View reviewed changes

HISTORY.md Show resolved Hide resolved

coatless reviewed Jun 22, 2021

View reviewed changes

mlpack-bot bot approved these changes Jun 23, 2021

View reviewed changes

mlpack-bot bot removed the s: needs review label Jun 23, 2021

Add source of GFG code.

8ac1887

coatless approved these changes Jun 23, 2021

View reviewed changes

zoq merged commit 86cf8f3 into mlpack:master Jun 23, 2021

jonpsy deleted the uniform branch June 23, 2021 18:35

This was referenced Jul 5, 2021

Release version 2.17.0: "25 Din Mein Paisa Double" #308

Closed

Release version 2.17.0: "Pachis Din Me Pesa Double" #310

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Das-Dennis weight initialization method #295

Add Das-Dennis weight initialization method #295

jonpsy commented Jun 18, 2021 •

edited

Loading

jonpsy Jun 18, 2021

jonpsy commented Jun 18, 2021 •

edited

Loading

jonpsy commented Jun 18, 2021 •

edited

Loading

zoq commented Jun 18, 2021

jonpsy commented Jun 18, 2021

zoq Jun 18, 2021

jonpsy Jun 18, 2021

zoq Jun 18, 2021

zoq Jun 18, 2021

jonpsy Jun 18, 2021

coatless Jun 22, 2021

jonpsy Jun 22, 2021

zoq Jun 23, 2021

jonpsy commented Jun 18, 2021

zoq commented Jun 18, 2021

jonpsy commented Jun 18, 2021 •

edited

Loading

jonpsy commented Jun 18, 2021

jonpsy commented Jun 19, 2021

zoq left a comment

coatless left a comment

coatless Jun 22, 2021

mlpack-bot bot left a comment

jonpsy commented Jun 23, 2021

coatless commented Jun 23, 2021

		point.insert_rows(point.n_rows, RowType(1).fill(
		delta * static_cast<ElemType>(beta)));

Add Das-Dennis weight initialization method #295

Add Das-Dennis weight initialization method #295

Conversation

jonpsy commented Jun 18, 2021 • edited Loading

Choose a reason for hiding this comment

jonpsy commented Jun 18, 2021 • edited Loading

jonpsy commented Jun 18, 2021 • edited Loading

zoq commented Jun 18, 2021

jonpsy commented Jun 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonpsy commented Jun 18, 2021

zoq commented Jun 18, 2021

jonpsy commented Jun 18, 2021 • edited Loading

jonpsy commented Jun 18, 2021

jonpsy commented Jun 19, 2021

zoq left a comment

Choose a reason for hiding this comment

coatless left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlpack-bot bot left a comment

Choose a reason for hiding this comment

jonpsy commented Jun 23, 2021

coatless commented Jun 23, 2021

jonpsy commented Jun 18, 2021 •

edited

Loading

jonpsy commented Jun 18, 2021 •

edited

Loading

jonpsy commented Jun 18, 2021 •

edited

Loading

jonpsy commented Jun 18, 2021 •

edited

Loading