Updating LightGBM Arguments #2948

singlis · 2019-03-13T22:49:04Z

This PR moves the LightGBM options from an all-in-one class to individual options for the binary, multiclass, regression and ranker trainers.

Options code that used to live in LightGbmArguments has moved to LightGbmBaseTrainer. Each trainer now has their own respective options class.
The booster options have changed a bit to hide the IBoosterFactory interface Make ISupportBoosterParameterFactory internal in LightGBM project #2559

each LightGBM trainer. This also hides the interface for the Booster Parameter Factory (IBoosterParameterFactory).

singlis · 2019-03-13T22:49:36Z

.vsts-dotnet-ci.yml

@@ -8,7 +8,7 @@ resources:
    image: microsoft/dotnet-buildtools-prereqs:centos-7-b46d863-20180719033416

  - container: UbuntuContainer
-    image: microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-10fcdcf-20190208200917
+    image: microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-mlnet-207e097-20190312152303


ignore this change... #Resolved

singlis · 2019-03-13T22:52:25Z

src/Microsoft.ML.LightGbm/LightGbmBinaryTrainer.cs

        internal LightGbmBinaryClassificationTrainer(IHostEnvironment env, Options options)
             : base(env, LoadNameValue, options, TrainerUtils.MakeBoolScalarLabel(options.LabelColumnName))
        {
+            Contracts.CheckUserArg(options.Sigmoid > 0, nameof(Options.Sigmoid), "must be > 0.");


This check should be on Multiclass and Ranking. #Resolved

singlis · 2019-03-13T22:56:02Z

src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

+        // Contains the passed in options when the API is called
+        private protected readonly TOptions LightGbmTrainerOptions;
+
+        // Contains the GBMOptions


GBMOptions [](start = 24, length = 10)

Ill move the argument below for this, I had rename to GbmOptions as it was conflicting with the Options class in the derived classes. #Resolved

singlis · 2019-03-13T22:56:57Z

src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

-            LightGbmTrainerOptions.FeatureColumnName = featureColumnName;
-            LightGbmTrainerOptions.ExampleWeightColumnName = exampleWeightColumnName;
-            LightGbmTrainerOptions.RowGroupColumnName = rowGroupColumnName;
+            LightGbmTrainerOptions = new TOptions


LightGbmTrainerOptions [](start = 12, length = 22)

will resolve this call the constructor below (via Options). #Resolved

singlis · 2019-03-13T23:13:37Z

src/Microsoft.ML.LightGbm/WrappedLightGbmInterface.cs

+        // Static override name map that maps friendly names to lightGBM arguments.
+        // If an argument is not here, then its name is identicaltto a lightGBM argument
+        // and does not require a mapping, for example, Subsample.
+        private static Dictionary<string, string> _nameMapping = new Dictionary<string, string>()


I dont like having a dictionary for this, even worse is having the dictionary defined here in this interface. If someone were to add new options to be handled in lightgbm, its not clear that they need to potentially update this as well in order for the correct lightgbm argument to get mapped. I think a better "option" (har har no pun intended) would be to have the Option classes build this dictionary, but I haven't worked on this yet and this at least gets things semi-working.
#Resolved

Ivanidzo4ka · 2019-03-13T23:17:06Z

test/BaselineOutput/Common/EntryPoints/core_ep-list.tsv

@@ -53,10 +53,17 @@ Trainers.FieldAwareFactorizationMachineBinaryClassifier	Train a field-aware fact
 Trainers.GeneralizedAdditiveModelBinaryClassifier	Trains a gradient boosted stump per feature, on all features simultaneously, to fit target values using least-squares. It mantains no interactions between features.	Microsoft.ML.Trainers.FastTree.Gam	TrainBinary	Microsoft.ML.Trainers.FastTree.GamBinaryClassificationTrainer+Options	Microsoft.ML.EntryPoints.CommonOutputs+BinaryClassificationOutput
 Trainers.GeneralizedAdditiveModelRegressor	Trains a gradient boosted stump per feature, on all features simultaneously, to fit target values using least-squares. It mantains no interactions between features.	Microsoft.ML.Trainers.FastTree.Gam	TrainRegression	Microsoft.ML.Trainers.FastTree.GamRegressionTrainer+Options	Microsoft.ML.EntryPoints.CommonOutputs+RegressionOutput
 Trainers.KMeansPlusPlusClusterer	K-means is a popular clustering algorithm. With K-means, the data is clustered into a specified number of clusters in order to minimize the within-cluster sum of squares. K-means++ improves upon K-means by using a better method for choosing the initial cluster centers.	Microsoft.ML.Trainers.KMeansTrainer	TrainKMeans	Microsoft.ML.Trainers.KMeansTrainer+Options	Microsoft.ML.EntryPoints.CommonOutputs+ClusteringOutput
+<<<<<<< HEAD


merge conflict
Just regenerate it locally and override #Closed

Ivanidzo4ka · 2019-03-13T23:17:43Z

test/Microsoft.ML.Core.Tests/UnitTests/TestEntryPoints.cs

@@ -225,7 +225,8 @@ private string GetBuildPrefix()
 #endif
        }

-        [Fact(Skip = "Execute this test if you want to regenerate the core_manifest and core_ep_list files")]
+        //[Fact(Skip = "Execute this test if you want to regenerate the core_manifest and core_ep_list files")]
+        [Fact]


Fact [](start = 9, length = 4)

don't forget to put it back. #Resolved

Ivanidzo4ka · 2019-03-13T23:35:46Z

src/Microsoft.ML.LightGbm/LightGbmMulticlassTrainer.cs

-            if (!Options.ContainsKey("metric"))
-                Options["metric"] = "multi_error";
+            if (!GbmOptions.ContainsKey("metric"))
+                GbmOptions["metric"] = "multi_error";


"multi_error" [](start = 39, length = 13)

we don't use metric enum at all? #Closed

Each trainer has its own respective metric enum and default metric type being assigned on initialization.

In reply to: 265372586 [](ancestors = 265372586)

Ivanidzo4ka · 2019-03-13T23:37:55Z

src/Microsoft.ML.LightGbm/LightGbmRankingTrainer.cs

            ch.CheckValue(groups, nameof(groups));
            // Add default metric.
-            if (!Options.ContainsKey("metric"))
-                Options["metric"] = "ndcg";
+            if (!GbmOptions.ContainsKey("metric"))


if (!GbmOptions.ContainsKey("metric")) [](start = 12, length = 38)

we don't have base metrics anymore, so we should provide enum in options and set it value here. #Closed

wschin · 2019-03-13T23:58:14Z

test/BaselineOutput/Common/EntryPoints/core_ep-list.tsv

 Trainers.LightGbmBinaryClassifier	Train a LightGBM binary classification model.	Microsoft.ML.Trainers.LightGbm.LightGbm	TrainBinary	Microsoft.ML.Trainers.LightGbm.Options	Microsoft.ML.EntryPoints.CommonOutputs+BinaryClassificationOutput
 Trainers.LightGbmClassifier	Train a LightGBM multi class model.	Microsoft.ML.Trainers.LightGbm.LightGbm	TrainMulticlass	Microsoft.ML.Trainers.LightGbm.Options	Microsoft.ML.EntryPoints.CommonOutputs+MulticlassClassificationOutput
 Trainers.LightGbmRanker	Train a LightGBM ranking model.	Microsoft.ML.Trainers.LightGbm.LightGbm	TrainRanking	Microsoft.ML.Trainers.LightGbm.Options	Microsoft.ML.EntryPoints.CommonOutputs+RankingOutput
 Trainers.LightGbmRegressor	LightGBM Regression	Microsoft.ML.Trainers.LightGbm.LightGbm	TrainRegression	Microsoft.ML.Trainers.LightGbm.Options	Microsoft.ML.EntryPoints.CommonOutputs+RegressionOutput
+>>>>>>> origin/master


Who are you? #Resolved

shmoradims · 2019-03-14T17:57:18Z

src/Microsoft.ML.LightGbm/LightGbmArguments.cs

-                /// If the tree partition step results in a leaf node with the sum of instance weight less than <see cref="MinimumChildWeight"/>,
-                /// the building process will give up further partitioning. In linear regression mode, this simply corresponds to minimum number
-                /// of instances needed to be in each node. The larger, the more conservative the algorithm will be.
-                /// </value>


what's happening to all these xml documentations? do you need to merge with master? Please make sure to move the xml docs when moving around the options. #Resolved

thanks Shahab - I captured these.

In reply to: 265696202 [](ancestors = 265696202)

wschin · 2019-03-15T00:30:58Z

src/Microsoft.ML.LightGbm/LightGbmArguments.cs

-    }
-
-    public interface IBoosterParameter
+    internal interface IBoosterParameterFactory : IComponentFactory<BoosterParameterBase>
    {


Should this be a best friend? I don't know if command line framework can call it now. #Closed

I can run the command line.

In reply to: 265815086 [](ancestors = 265815086)

wschin · 2019-03-15T00:32:32Z

src/Microsoft.ML.LightGbm/LightGbmArguments.cs

-
-                    res[GetOptionName(field.Name)] = field.GetValue(BoosterParameterOptions);
-                }
+                return BuildOptions();
            }


Not your problem. C# doesn't work well here, so we must have two (conceptually) identical implementations..

thanks for mentioning this, I will keep this active for other reviewers.

In reply to: 265815338 [](ancestors = 265815338)

wschin · 2019-03-15T00:33:59Z

src/Microsoft.ML.LightGbm/LightGbmArguments.cs

-            {
-                base.UpdateParameters(res);
-                res["boosting_type"] = Name;
+                res[LightGbmInterfaceUtils.GetOptionName(field.Name)] = field.GetValue(BoosterOptions);
            }


Do we need to check if key LightGbmInterfaceUtils.GetOptionName(field.Name) exists as this function is called Update? #Resolved

So LightGbmInterfaceUtils.GetOptionName does not have a dictionary that it works from -- it just converts the field name to lower case plus some underscoring. So there should be no need to check for a key. There could already be a key in the res dictionary - but thats ok as we are overriding it and like you said, this is an Update function.

In reply to: 265815549 [](ancestors = 265815549)

Ill mark this as pending as I want to make sure I answered your question.

In reply to: 266178686 [](ancestors = 266178686,265815549)

Ivanidzo4ka · 2019-03-15T21:10:22Z

src/Microsoft.ML.LightGbm/LightGbmRankingTrainer.cs

+                None,
+                Default,
+                Map,
+                Ndcg


Ndcg [](start = 16, length = 4)

NormalizedDiscountedCumulativeGain or Lambdarank #Closed

Ivanidzo4ka · 2019-03-15T21:10:35Z

src/Microsoft.ML.LightGbm/LightGbmRankingTrainer.cs

+            {
+                None,
+                Default,
+                Map,


Map [](start = 16, length = 3)

MeanAveragePrecision #Closed

Ivanidzo4ka · 2019-03-15T21:12:25Z

src/Microsoft.ML.LightGbm/LightGbmRankingTrainer.cs

+            /// </summary>
+            [Argument(ArgumentType.AtMostOnce, HelpText = "Comma seperated list of gains associated to each relevance label.", ShortName = "gains")]
+            [TGUI(Label = "Ranking Label Gain")]
+            public string CustomGains = "0,3,7,15,31,63,127,255,511,1023,2047,4095";


string CustomGains [](start = 19, length = 18)

public int[] CustomGains = new int[] { 0, 3, 7, 15, 31,63,... }; #Closed

Ivanidzo4ka · 2019-03-15T21:13:03Z

src/Microsoft.ML.LightGbm/LightGbmMulticlassTrainer.cs

+                None,
+                Default,
+                Merror,
+                Mlogloss,


Mlogloss [](start = 16, length = 8)

LogLoss #Closed

Ivanidzo4ka · 2019-03-15T21:14:29Z

src/Microsoft.ML.LightGbm/LightGbmMulticlassTrainer.cs

+            {
+                None,
+                Default,
+                Merror,


Merror [](start = 16, length = 6)

Error #Closed

codecov · 2019-03-15T21:36:03Z

Codecov Report

❗ No coverage uploaded for pull request base (master@0831865). Click here to learn what that means.
The diff coverage is 86.96%.

@@            Coverage Diff            @@
##             master    #2948   +/-   ##
=========================================
  Coverage          ?   72.35%           
=========================================
  Files             ?      803           
  Lines             ?   143388           
  Branches          ?    16154           
=========================================
  Hits              ?   103750           
  Misses            ?    35214           
  Partials          ?     4424

Flag	Coverage Δ
#Debug	`72.35% <86.96%> (?)`
#production	`68.07% <86.34%> (?)`
#test	`88.52% <100%> (?)`

Impacted Files	Coverage Δ
...ML.LightGbm.StaticPipe/LightGbmStaticExtensions.cs	`45.45% <0%> (ø)`
...cenariosWithDirectInstantiation/TensorflowTests.cs	`90.81% <100%> (ø)`
...est/Microsoft.ML.Predictor.Tests/TestPredictors.cs	`63.8% <100%> (ø)`
.../Microsoft.ML.LightGbm/WrappedLightGbmInterface.cs	`83.72% <100%> (ø)`
src/Microsoft.ML.LightGbm/LightGbmBinaryTrainer.cs	`87.71% <100%> (ø)`
src/Microsoft.ML.LightGbm/LightGbmCatalog.cs	`100% <100%> (ø)`
...Microsoft.ML.LightGbm/LightGbmMulticlassTrainer.cs	`89.57% <100%> (ø)`
test/Microsoft.ML.TestFramework/Learners.cs	`90.74% <100%> (ø)`
...osoft.ML.Tests/TrainerEstimators/TreeEstimators.cs	`98.52% <100%> (ø)`
src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs	`62.44% <75.58%> (ø)`
... and 3 more

singlis · 2019-03-16T01:16:00Z

test/BaselineOutput/Common/LightGBMR/LightGBMReg-CV-generatedRegressionDataset-out.txt

@@ -35,10 +35,10 @@ Virtual memory usage(MB): %Number%
 [1] 'Loading data for LightGBM' started.
 [1] 'Loading data for LightGBM' finished in %Time%.
 [2] 'Training with LightGBM' started.
-[2] (%Time%)	Iteration: 50	Training-l2: 37.107605006517
+[2] (%Time%)	Iteration: 50	Training-rmse: 6.09160118577349


rmse [](start = 36, length = 4)

I am investigating the delta in these numbers. I believe it has to do with the metric type we were previously passing vs what is passed now. Previously we were setting the metric type to l2 which is incorrect for rmse. Now we are passing l2_root.
#Resolved

Confirmed this is due to changing the code to use l2_root(aka rmse), so this change is expected.

In reply to: 266183353 [](ancestors = 266183353)

t # .vsts-dotnet-ci.yml

Ivanidzo4ka · 2019-03-18T22:17:55Z

src/Microsoft.ML.LightGbm/LightGbmArguments.cs

+            [TlcModule.SweepableDiscreteParam("RegAlpha", new object[] { 0f, 0.5f, 1f })]
+            public double L1Regularization = 0;
+
+            BoosterParameterBase IComponentFactory<BoosterParameterBase>.CreateComponent(IHostEnvironment env)
            {


{ [](start = 12, length = 1)

nit: you can just BoosterParameterBase IComponentFactory<BoosterParameterBase>.CreateComponent(IHostEnvironment env) => BuildOptions(); #Resolved

Ivanidzo4ka · 2019-03-18T22:20:52Z

src/Microsoft.ML.LightGbm/LightGbmArguments.cs

+            Contracts.CheckUserArg(options.SubsampleFraction > 0 && options.SubsampleFraction <= 1, nameof(Options.SubsampleFraction), "must be in (0,1].");
+            Contracts.CheckUserArg(options.FeatureFraction > 0 && options.FeatureFraction <= 1, nameof(Options.FeatureFraction), "must be in (0,1].");
+            Contracts.CheckUserArg(options.L2Regularization >= 0, nameof(Options.L2Regularization), "must be >= 0.");
+            Contracts.CheckUserArg(options.L1Regularization >= 0, nameof(Options.L1Regularization), "must be >= 0.");


Don't you need this validation across all boosters? #Resolved

Yes - these are all in the BoosterParameterBase -- so GradientBooster, DartBooster and GossBooster all share these arguments. Therefore I added these restrictions. These restrictions come from the lightgbm documentation.

In reply to: 266665035 [](ancestors = 266665035)

But you run validation only in GradientBooster DartBooster doesn't check this options as far as I can see.

In reply to: 266672503 [](ancestors = 266672503,266665035)

You are right - I was thinking DartBooster derives from Gradient, but it doesnt (*it used to :))

In reply to: 266673967 [](ancestors = 266673967,266672503,266665035)

Ivanidzo4ka · 2019-03-18T22:21:27Z

src/Microsoft.ML.LightGbm/LightGbmArguments.cs

-        /// </value>
-        [Argument(ArgumentType.AtMostOnce, HelpText = "Printing running messages.")]
-        public bool Silent = true;
+        public GradientBooster(Options options)


GradientBooster [](start = 15, length = 15)

Constuctor of DartBooster is internal #Resolved

So that is actually intentional since the user doesnt construct a DartBooster, they construct a DartBooster.Options. However, you do point out that GradientBooster is public -- so I made that internal.

In reply to: 266665204 [](ancestors = 266665204)

Ivanidzo4ka · 2019-03-18T22:23:18Z

src/Microsoft.ML.LightGbm/LightGbmBinaryTrainer.cs

        {
        }

        private protected override CalibratedModelParametersBase<LightGbmBinaryModelParameters, PlattCalibrator> CreatePredictor()
        {
            Host.Check(TrainedEnsemble != null, "The predictor cannot be created before training is complete");
-            var innerArgs = LightGbmInterfaceUtils.JoinParameters(Options);
+            var innerArgs = LightGbmInterfaceUtils.JoinParameters(base.GbmOptions);


base [](start = 66, length = 4)

Does base necessary? #Resolved

no that comes from the VS renaming....I will remove.

In reply to: 266665665 [](ancestors = 266665665)

Ivanidzo4ka

wschin · 2019-03-18T22:46:53Z

src/Microsoft.ML.LightGbm/LightGbmBinaryTrainer.cs

@@ -82,7 +85,7 @@ private static IPredictorProducing<float> Create(IHostEnvironment env, ModelLoad
    /// The <see cref="IEstimator{TTransformer}"/> for training a boosted decision tree binary classification model using LightGBM.
    /// </summary>
    /// <include file='doc.xml' path='doc/members/member[@name="LightGBM_remarks"]/*' />
-    public sealed class LightGbmBinaryClassificationTrainer : LightGbmTrainerBase<float,
+    public sealed class LightGbmBinaryClassificationTrainer : LightGbmTrainerBase<LightGbmBinaryClassificationTrainer.Options,float,


missing space #Resolved

wschin · 2019-03-18T22:48:12Z

src/Microsoft.ML.LightGbm/LightGbmBinaryTrainer.cs

        internal LightGbmBinaryClassificationTrainer(IHostEnvironment env, Options options)
             : base(env, LoadNameValue, options, TrainerUtils.MakeBoolScalarLabel(options.LabelColumnName))
        {
+            Contracts.CheckUserArg(options.Sigmoid > 0, nameof(Options.Sigmoid), "must be > 0.");
+            Contracts.CheckUserArg(options.WeightOfPositiveExamples > 0, nameof(Options.WeightOfPositiveExamples), "must be >= 0.");


= [](start = 124, length = 2)

#Resolved

nice catch

In reply to: 266671856 [](ancestors = 266671856)

wschin · 2019-03-18T22:49:13Z

src/Microsoft.ML.LightGbm/LightGbmBinaryTrainer.cs

@@ -144,10 +226,7 @@ private protected override void CheckDataValid(IChannel ch, RoleMappedData data)

        private protected override void CheckAndUpdateParametersBeforeTraining(IChannel ch, RoleMappedData data, float[] labels, int[] groups)
        {


One-line function uses => #Resolved

wschin · 2019-03-18T22:52:50Z

src/Microsoft.ML.LightGbm/LightGbmMulticlassTrainer.cs

+                NameMapping.Add(nameof(EvaluateMetricType.Error), "multi_error");
+                NameMapping.Add(nameof(EvaluateMetricType.LogLoss), "multi_logloss");
+            }
+


This might not work with new Options(). #Resolved

Since this is static, it will be created before our main function is called.

In reply to: 266672901 [](ancestors = 266672901)

wschin · 2019-03-18T22:53:23Z

src/Microsoft.ML.LightGbm/LightGbmMulticlassTrainer.cs

+                  {
+                    LabelColumnName = labelColumnName,
+                    FeatureColumnName = featureColumnName,
+                    ExampleWeightColumnName = exampleWeightColumnName,


ExampleWeightColumnName [](start = 20, length = 23)

WeightColumnName #Resolved

I think rename should happen in a different PR.

In reply to: 266673053 [](ancestors = 266673053)

I think rename should happen in a different PR.
+1

In reply to: 266674022 [](ancestors = 266674022,266673053)

wschin · 2019-03-18T22:54:20Z

src/Microsoft.ML.LightGbm/LightGbmRankingTrainer.cs

+            /// <summary>
+            /// Comma-separated list of gains associated with each relevance label.
+            /// </summary>
+            [Argument(ArgumentType.AtMostOnce, HelpText = "Comma seperated list of gains associated to each relevance label.", ShortName = "gains")]


Comma seperated list [](start = 59, length = 20)

It is array now. #Resolved

wschin · 2019-03-18T23:03:56Z

src/Microsoft.ML.LightGbm/WrappedLightGbmInterface.cs

@@ -225,6 +226,27 @@ public static string JoinParameters(Dictionary<string, object> parameters)
            return string.Join(" ", res);
        }

+        public static string GetOptionName(string name)
+        {


Doc string please. #Resolved

wschin

I only left some minor comments. Thank you!

singlis added 7 commits March 7, 2019 13:52

Initial checkin

fe3b078

update...

fd29ea2

Update..

c25cde7

updating...

df75991

Breaking down the LightGBM options class into separate option class for

355241f

each LightGBM trainer. This also hides the interface for the Booster Parameter Factory (IBoosterParameterFactory).

Merge remote-tracking branch 'origin/master' into singlis/options

f252d9b

- updating code from namespace changes

6b54dfe

singlis requested review from codemzs, Ivanidzo4ka, wschin, TomFinley, ganik and sfilipi March 13, 2019 22:49

singlis self-assigned this Mar 13, 2019

singlis commented Mar 13, 2019

View reviewed changes

Merge remote-tracking branch 'origin/master' into singlis/options

5182721

singlis commented Mar 13, 2019

View reviewed changes

Ivanidzo4ka reviewed Mar 13, 2019

View reviewed changes

wschin reviewed Mar 13, 2019

View reviewed changes

resolving merge conflict

8504317

shmoradims reviewed Mar 14, 2019

View reviewed changes

wschin reviewed Mar 15, 2019

View reviewed changes

Ivanidzo4ka reviewed Mar 15, 2019

View reviewed changes

singlis commented Mar 16, 2019

View reviewed changes

- Updating based upon feedback, updated test baseline files.

c930cc1

singlis changed the title ~~WIP: Updating LightGBM Arguments~~ Updating LightGBM Arguments Mar 18, 2019

singlis added 3 commits March 18, 2019 14:32

Merge remote-tracking branch 'origin/master' into singlis/options

31533d2

t # .vsts-dotnet-ci.yml

Removing merge conflict

ec46829

updating...

b1c21fd

Ivanidzo4ka reviewed Mar 18, 2019

View reviewed changes

Ivanidzo4ka approved these changes Mar 18, 2019

View reviewed changes

wschin reviewed Mar 18, 2019

View reviewed changes

wschin approved these changes Mar 18, 2019

View reviewed changes

singlis added 2 commits March 18, 2019 16:13

- Updating based on feedback

5cdace2

Merge remote-tracking branch 'origin/master' into singlis/options

241a58a

singlis merged commit aea88dc into dotnet:master Mar 19, 2019

singlis mentioned this pull request Apr 1, 2019

Check range of ScalePosWeight in LightGBM #2147

Closed

ghost locked as resolved and limited conversation to collaborators Mar 23, 2022

		@@ -144,10 +226,7 @@ private protected override void CheckDataValid(IChannel ch, RoleMappedData data)

		private protected override void CheckAndUpdateParametersBeforeTraining(IChannel ch, RoleMappedData data, float[] labels, int[] groups)
		{

Updating LightGBM Arguments #2948

Updating LightGBM Arguments #2948

Uh oh!

Conversation

singlis commented Mar 13, 2019

Uh oh!

singlis Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singlis Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singlis Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singlis Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singlis Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Mar 13, 2019 • edited by singlis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Mar 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wschin Mar 13, 2019 • edited by singlis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shmoradims Mar 14, 2019 • edited by singlis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wschin Mar 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wschin Mar 15, 2019 • edited by singlis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Mar 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ivanidzo4ka Mar 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singlis Mar 13, 2019 •

edited

Loading

singlis Mar 13, 2019 •

edited

Loading

singlis Mar 13, 2019 •

edited

Loading

singlis Mar 13, 2019 •

edited

Loading

singlis Mar 13, 2019 •

edited

Loading

Ivanidzo4ka Mar 13, 2019 •

edited

Loading

Ivanidzo4ka Mar 13, 2019 •

edited by singlis

Loading

Ivanidzo4ka Mar 13, 2019 •

edited

Loading

Ivanidzo4ka Mar 13, 2019 •

edited

Loading

wschin Mar 13, 2019 •

edited by singlis

Loading

shmoradims Mar 14, 2019 •

edited by singlis

Loading

wschin Mar 15, 2019 •

edited

Loading

wschin Mar 15, 2019 •

edited by singlis

Loading

Ivanidzo4ka Mar 15, 2019 •

edited

Loading

Ivanidzo4ka Mar 15, 2019 •

edited

Loading

Ivanidzo4ka Mar 15, 2019 •

edited

Loading

Ivanidzo4ka Mar 15, 2019 •

edited

Loading

Ivanidzo4ka Mar 15, 2019 •

edited

Loading

codecov bot commented Mar 15, 2019 •

edited

Loading

singlis Mar 16, 2019 •

edited

Loading

singlis Mar 18, 2019 •

edited

Loading

Ivanidzo4ka Mar 18, 2019 •

edited by singlis

Loading

Ivanidzo4ka Mar 18, 2019 •

edited by singlis

Loading

Ivanidzo4ka Mar 18, 2019 •

edited by singlis

Loading

Ivanidzo4ka Mar 18, 2019 •

edited by singlis

Loading