Adding training statistics for LR in the HAL learners package. #1392

sfilipi · 2018-10-26T06:56:32Z

To address #612 maybe we can introduce another LR trainer in the HAL learners package.

Heresy #1 - it derives from LR making this last one not sealed, and exposing more of its fields.
Heresy #2 - the only thing this learner does differently from LR is to compute the std in the training stats are requested.

I think afa usability this is more accessible than just having a utility method that takes the LR trainer/predictor and recomputes the stats with std.

Thought?

@GalOshri @TomFinley @eerhardt

…mpute the std errors

sfilipi · 2018-10-26T06:57:56Z

src/Microsoft.ML.HalLearners/LRWithTrainingStatistics.cs

+            Double priorPosRate = PosWeight / WeightSum;
+            Contracts.Assert(0 <= priorPosRate && priorPosRate <= 1);
+            float nullDeviance = (priorPosRate <= 1e-15 || 1 - priorPosRate <= 1e-15) ?
+                0f : (float)(2 * WeightSum * MathUtils.Entropy(priorPosRate, true));


if we go down this route, this will need to get refactored to avoid repetition. Doing it the fast way to see if it even passes the smell test :)
) #Resolved

eerhardt · 2018-10-26T14:49:38Z

src/Microsoft.ML.HalLearners/LRWithTrainingStatistics.cs

+
+    /// <include file='doc.xml' path='doc/members/member[@name="LBFGS"]/*' />
+    /// <include file='doc.xml' path='docs/members/example[@name="LogisticRegressionBinaryClassifier"]/*' />
+    public sealed partial class LogisticRegressionWithStats : LogisticRegression


(nit) In general I think the file name should match the class name. Here we have LogisticRegressionWithStats vs LRWithTrainingStatistics.cs #Resolved

GalOshri · 2018-10-26T20:24:33Z

My initial thought is that it might be confusing to add a completely separate trainer that is exactly the same but just has one additional capability of providing the stats.

Maybe I just don't have a good idea of what is the alternative solution. Would a utility method be called after training? Or do I have to specify it before training? Conceptually, a user first trains the model, then wants to look up the stats.

What do others think? #Resolved

sfilipi · 2018-10-26T20:28:16Z

My initial thought is that it might be confusing to add a completely separate trainer that is exactly the same but just has one additional capability of providing the stats.
Maybe I just don't have a good idea of what is the alternative solution. Would a utility method be called after training? Or do I have to specify it before training? Conceptually, a user first trains the model, then wants to look up the stats.
What do others think?

If we don't worry about the cmd usage, the utility method is fine. #Resolved

…., rather than a separate trainer.

TomFinley

Hi @sfilipi thanks for working on this. I am not a fan of this. Everywhere else we have been an advocation that the place where this sort of "side information" would be published would be gotten by the transformer that is returned by this Fit method -- this indeed is the reason why we bothered to make the IEstimator strongly typed.

And this at least conceptually appears to work pretty well, and I'm not sure I see too many flaws in it. When you fit a term estimator, you could get the dictionary from the transform. When you fit a normalizer, you could get the normalizer scaling/offsets from the normalizer transform. When you train a GBDT, you get the tree from the predictor in the transform. When you train a linear model, you can get the linear terms from the predictor in the transform. And even with training stats, we already have OLS, and that again publishes its statistics in the predictor in the transform.

I would really like it if you could follow the same pattern here. Making IEstimators contain information about the model is completely against the spirit of IEstimator and ITransformer. ITransformer is the model, an IEstimator is a way to make models. They cannot contain model information.

On the whole by doing this I think you will also find the odd architectural decisions you thought you were forced into (the so-called "heresies," which should have been a clue that something seriously wrong was going on) will go away.

…edictor.

klausmh · 2018-10-31T23:19:29Z

From a practical point of view, my impression is that in order to get training statistics from logistic regression, you would have to use MKL. This would then require you to add a rather large assembly to your project. At least in my use case, this may be prohibitive. #Pending

sfilipi · 2018-11-01T04:59:06Z

hi @klausmh, what is the bar for the size addition? We don't use all of MKL, we have a custom build of it that contains only functions that are needed by ML.Net. The whole nuget is about 320MB with libraries for all the platforms, win64. win 86, linux and mac. Windows debug is only 70MB.

In reply to: 434880120 [](ancestors = 434880120)

klausmh · 2018-11-01T18:13:12Z

In reply to: 434880120 [](ancestors = 434880120)

In our case, this would be added to a desktop installer that is currently 262 MB, so it would be a 25% increase in installer size. I would think that something exceeding 10 MB would be problematic (in this specific case). #Resolved

wschin · 2018-11-05T19:07:45Z

test/Microsoft.ML.Tests/TrainerEstimators/LbfgsTests.cs

+
+            LinearModelStatistics.TryGetBiasStatistics(stats, 2, out stdError, out zScore, out pValue);
+
+            Assert.True(stdError > 0);


May we put a more strict range for each statistic? #Resolved

wschin · 2018-11-05T19:09:05Z

src/Microsoft.ML.HalLearners/LogisticRegressionTrainingStats.cs

+    public static class LogisticRegressionTrainingStats
+    {
+
+        public static void ComputeExtendedTrainingStatistics(this LinearBinaryPredictor model, IChannel ch, float l2Weight = LogisticRegression.Arguments.Defaults.L2Weight)


The function name can be just compute(...) because it's inside LogisticRegressionTrainingStats. #Resolved

wschin · 2018-11-05T19:49:14Z

src/Microsoft.ML.HalLearners/LogisticRegressionTrainingStats.cs

+                // Iterate through all entries of inverse Hessian to make adjustment to variance.
+                // A discussion on ridge regularized LR coefficient covariance matrix can be found here:
+                // http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3228544/
+                // http://www.inf.unibz.it/dis/teaching/DWDM/project2010/LogisticRegression.pdf


Suggested change

// http://www.inf.unibz.it/dis/teaching/DWDM/project2010/LogisticRegression.pdf

// http://www.inf.unibz.it/dis/teaching/DWDM/project2010/LogisticRegression.pdf (Section "Significance testing in ridge logistic regression")

``` #Resolved

wschin · 2018-11-05T21:22:15Z

src/Microsoft.ML.HalLearners/LogisticRegressionTrainingStats.cs

+                    for (int iCol = 0; iCol <= iRow; iCol++)
+                    {
+                        var entry = (Single)invHessian[ioffset];
+                        var adjustment = -l2Weight * entry * entry;


This line doesn't make a lot of sense to me. The code first compute \sigma=sqrt(x+\lambda) and them do something like \sigma + \lambda * \sigma^2. #Resolved

I'll review it too. Thanks for pointing out.

In reply to: 230916692 [](ancestors = 230916692)

wschin · 2018-11-05T21:31:57Z

src/Microsoft.ML.HalLearners/LogisticRegressionTrainingStats.cs

+            {
+                // Iterate through all entries of inverse Hessian to make adjustment to variance.
+                // A discussion on ridge regularized LR coefficient covariance matrix can be found here:
+                // http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3228544/


I really don't like putting a very old doc inline and asking someone to read the whole article.

Suggested change

// http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3228544/

// http://www.aloki.hu/pdf/0402_171179.pdf (Equations 11 and 25)

``` #Resolved

TomFinley

Looks better @sfilipi thanks for addressing. I'm not sure how to clear my "request changes" review status so approving. An improvement for sure, though not perhaps quite what is needed by the people.

…ions, one the old MKl way in the HAL Learners package, and the other making use of Math.Numerics

sfilipi · 2018-11-07T08:33:39Z

test/Microsoft.ML.Tests/TrainerEstimators/LbfgsTests.cs

@@ -38,5 +39,55 @@ public void TestEstimatorPoissonRegression()
            TestEstimatorCore(pipe, dataView);
            Done();
        }
+
+        [Fact]
+        public void TestLogisticRegressionStats()


TestLogisticRegressionStats [](start = 20, length = 27)

combine both tests together #WontFix

Currently running into issues with this. Will investigate and log a bug.

In reply to: 231416451 [](ancestors = 231416451)

eerhardt · 2018-11-08T20:53:58Z

src/Microsoft.ML.StandardLearners/Microsoft.ML.StandardLearners.csproj


  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <IncludeInPackage>Microsoft.ML</IncludeInPackage>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
  </PropertyGroup>

+  <ItemGroup>
+    <PackageReference Include="MathNet.Numerics.Signed" Version="$(MathNumericPackageVersion)" />


note: you'll need to add this PackageReference to our NuGet package as well:

machinelearning/pkg/Microsoft.ML/Microsoft.ML.nupkgproj

Lines 11 to 17 in f222025

<PackageReference Include="Newtonsoft.Json" Version="$(NewtonsoftJsonPackageVersion)" />

<PackageReference Include="System.Reflection.Emit.Lightweight" Version="$(SystemReflectionEmitLightweightPackageVersion)" />

<PackageReference Include="System.Threading.Tasks.Dataflow" Version="$(SystemThreadingTasksDataflowPackageVersion)" />

<PackageReference Include="System.CodeDom" Version="$(SystemCodeDomPackageVersion)" />

<PackageReference Include="System.Memory" Version="$(SystemMemoryVersion)" />

<PackageReference Include="System.Collections.Immutable" Version="$(SystemCollectionsImmutableVersion)" />

<PackageReference Include="System.ComponentModel.Composition" Version="$(SystemComponentModelCompositionVersion)" />

#Resolved

sfilipi · 2018-11-09T07:49:17Z

@TomFinley most things are changed from your approval, if you wanted to take another look.

sfilipi · 2018-11-09T07:51:09Z

src/Microsoft.ML.StandardLearners/Standard/LogisticRegression/LogisticRegression.cs

+            /// More than 1000 weights might take a few minutes. For those cases consider using the instance of <see cref="IComputeLRTrainingStd"/>
+            /// present in the Microsoft.ML.HalLearners package. That computes the statistics using hardware acceleration.
+            /// </summary>
+            public IComputeLRTrainingStd StdComputer;


StdComputer; [](start = 40, length = 13)

@tfinley@gmail.com the analyzer complained if i initialize here, and suggest to use the ctor instead. Is that legit? Asking since most of our Arguments are initialized here.

…rning-1 into trainingStats

eerhardt · 2018-11-12T15:28:53Z

build/Dependencies.props

@@ -22,6 +22,7 @@
    <SystemIOFileSystemAccessControl>4.5.0</SystemIOFileSystemAccessControl>
    <SystemSecurityPrincipalWindows>4.5.0</SystemSecurityPrincipalWindows>
    <TensorFlowVersion>1.10.0</TensorFlowVersion>
+    <MathNumericPackageVersion>4.6.0</MathNumericPackageVersion>


(nit) since this is a dependency in the MIcrosoft.ML nuget package, it belongs under the Core Product Dependencies section above. #Resolved

eerhardt · 2018-11-12T15:33:33Z

src/Microsoft.ML.StandardLearners/Standard/ModelStatistics.cs

-using System.ComponentModel;
-using System.IO;
-using System.Linq;
+using MathNet.Numerics.LinearAlgebra;


Is this using necessary? I don't see any Math.NET usages below. #Resolved

eerhardt · 2018-11-12T15:34:24Z

src/Microsoft.ML.StandardLearners/Standard/LogisticRegression/LogisticRegression.cs

@@ -397,4 +426,113 @@ public static CommonOutputs.BinaryClassificationOutput TrainBinary(IHostEnvironm
                () => LearnerEntryPointsUtils.FindColumn(host, input.TrainingData.Schema, input.WeightColumn));
        }
    }
+
+    /// <summary>
+    /// Computes the standart deviation matrix of each of the non-zero training weights, needed to calculate further the standart deviation,


standart deviation [](start = 21, length = 18)

standard deviation ? #Resolved

Here and in a few places.

In reply to: 232703545 [](ancestors = 232703545)

eerhardt · 2018-11-12T15:40:32Z

src/Microsoft.ML.StandardLearners/Standard/LogisticRegression/LogisticRegression.cs

+            double[,] matrixHessian = new double[numSelectedParams, numSelectedParams];
+
+            int hessianLength = 0;
+            int dimention = numSelectedParams - 1;


dimention => dimension. #Resolved

eerhardt · 2018-11-12T15:42:47Z

src/Microsoft.ML.StandardLearners/Standard/LogisticRegression/LogisticRegression.cs

+                }
+            }
+
+            var h = Matrix<double>.Build.DenseOfArray(matrixHessian);


I must be missing something here, so please forgive my ignorance.

What exactly are we using Math.NET for in this code? Are we just using it for the Matrix type, and for Inversing a Matrix? Or is there something else we are using that I'm not seeing?

If it is just for the Matrix type and Inverse, I don't think a full dependency on Math.NET is necessary. #Pending

Not missing anything. Math.Net numeric is used just to calculate the inverse. We discussed the option to implement it ourselves, and it was deemed error prone to round-of errors and unnecessary to own that code.
There is also a parallel pending discussion on weather we need to have an alternative method to MKL, which seems like it is leaning towards just keeping the MKL version.

In reply to: 232706971 [](ancestors = 232706971)

OK, thanks for the information.

eerhardt · 2018-11-12T15:48:24Z

src/Microsoft.ML.HalLearners/LogisticRegressionTrainingStats.cs

+{
+    using Mkl = OlsLinearRegressionTrainer.Mkl;
+
+    public sealed class ComputeLRTrainingStdThroughHal : IComputeLRTrainingStd


(nit) the file name and the class name should match, so it is easier to find the class on the file system. #Resolved

eerhardt · 2018-11-12T15:48:38Z

src/Microsoft.ML.StandardLearners/Standard/LogisticRegression/LogisticRegression.cs

+            {
+                // Iterate through all entries of inverse Hessian to make adjustment to variance.
+                // A discussion on ridge regularized LR coefficient covariance matrix can be found here:
+                // http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3228544/


How come this link is different than the link in the link in the ComputeLRTrainingStdThroughHal class? #Resolved

eerhardt · 2018-11-12T15:49:18Z

src/Microsoft.ML.StandardLearners/Standard/LogisticRegression/LogisticRegression.cs

+
+            if (l2Weight > 0)
+            {
+                // Iterate through all entries of inverse Hessian to make adjustment to variance.


Maybe extract this whole chunk of code into an internal utility method so it doesn't need to be duplicated between here and in ComputeLRTrainingStdThroughHal. #Resolved

…rning-1 into trainingStats

eerhardt · 2018-11-12T18:15:28Z

src/Microsoft.ML.StandardLearners/Standard/LogisticRegression/LogisticRegression.cs

    /// p-value and z-Score.
    /// If you need fast calculations, use the <see cref="IComputeLRTrainingStd"/> implementation in the Microsoft.ML.HALLearners package,
    /// which makes use of hardware acceleration.
    /// Due to the existence of regularization, an approximation is used to compute the variances of the trained linear coefficients.
    /// </summary>
-    public interface IComputeLRTrainingStd
+    public abstract class IComputeLRTrainingStd


We shouldn't be naming a class ISomething. I is the prefix for interfaces. #Resolved

totally!

In reply to: 232762008 [](ancestors = 232762008)

sfilipi · 2018-11-13T00:45:45Z

@wschin did you have any more comments for this PR? Does it look good to go?

eerhardt

I'm not excited about making our core NuGet package dependent on MathNet.Numerics.Signed, especially for 1 function (inversing a matrix). But I won't stand in your way of merging this PR.

sfilipi · 2018-11-13T04:27:44Z

Thanks @eerhardt and @TomFinley. Based on customer needs, and the evaluation of benchmarks, it is likely that i turn around and remove the dependency. I am not thrilled either.

eerhardt · 2018-11-13T15:19:39Z

One option we could do (now that I'm thinking a bit more clearly) is now that we have the extensibility in place, any consumer can now derive their own ComputeLRTrainingStd implementation themselves. The consumer can take the dependency on MathNet.Numerics and copy the ComputeLRTrainingStdImpl implementation into their code. They can supply their implementation through Args.StdComputer.

Then we can remove the direct dependency from Microsoft.ML, and every customer of ML.NET doesn't need to take this dependency.

Thoughts?

Adding another flavor of LR in the HAL learners package. This will co…

2d5fbf6

…mpute the std errors

sfilipi requested review from TomFinley, Ivanidzo4ka and eerhardt October 26, 2018 06:56

sfilipi commented Oct 26, 2018

View reviewed changes

sfilipi mentioned this pull request Oct 26, 2018

Training Statistics for linear learners. #612

Closed

eerhardt reviewed Oct 26, 2018

View reviewed changes

GalOshri mentioned this pull request Oct 29, 2018

LinearPredictor.Statistics is always null #1006

Closed

Adding ComputeExtendedTrainingStatistics as an extension method on LR…

68dd4a6

…., rather than a separate trainer.

sfilipi requested a review from Zruty0 October 30, 2018 23:33

adding test checks

e8fede2

sfilipi changed the title ~~WIP: Adding another flavor of LR in the HAL learners package.~~ Adding training statistics for LR in the HAL learners package. Oct 31, 2018

TomFinley suggested changes Oct 31, 2018

View reviewed changes

sfilipi added 2 commits October 31, 2018 12:47

merging from master

f649287

moving the calculations of the extended training statistics in the pr…

06f9704

…edictor.

wschin reviewed Nov 5, 2018

View reviewed changes

TomFinley reviewed Nov 6, 2018

View reviewed changes

TomFinley approved these changes Nov 6, 2018

View reviewed changes

sfilipi added 2 commits November 7, 2018 00:31

merging from master

96127d0

Creating two separate methods to compute the matrix of standartDeviat…

3831f5d

…ions, one the old MKl way in the HAL Learners package, and the other making use of Math.Numerics

sfilipi commented Nov 7, 2018

View reviewed changes

eerhardt reviewed Nov 8, 2018

View reviewed changes

addressing PR comments

fe29307

sfilipi commented Nov 9, 2018

View reviewed changes

sfilipi added 6 commits November 9, 2018 10:08

Merge branch 'master' into trainingStats

f0b4707

baselines have more information because of the added calculations

5386a8c

more baseline updates.

fb897ed

Merge branch 'master' into trainingStats

377b462

post merge test fixes.

89301b4

Merge branch 'trainingStats' of https://github.com/sfilipi/machinelea…

46316ea

…rning-1 into trainingStats

eerhardt reviewed Nov 12, 2018

View reviewed changes

sfilipi added 2 commits November 12, 2018 09:50

Eric's comments

c8d060a

Merge branch 'trainingStats' of https://github.com/sfilipi/machinelea…

737d173

…rning-1 into trainingStats

eerhardt reviewed Nov 12, 2018

View reviewed changes

renaming interface.

39ca55e

eerhardt approved these changes Nov 13, 2018

View reviewed changes

sfilipi merged commit 0c60374 into dotnet:master Nov 13, 2018

sfilipi deleted the trainingStats branch November 15, 2018 18:44

eerhardt mentioned this pull request Feb 24, 2021

Added initial design doc for cross platform #5628

Closed

ghost locked as resolved and limited conversation to collaborators Mar 27, 2022


		LinearModelStatistics.TryGetBiasStatistics(stats, 2, out stdError, out zScore, out pValue);

		Assert.True(stdError > 0);

	// http://www.inf.unibz.it/dis/teaching/DWDM/project2010/LogisticRegression.pdf
	// http://www.inf.unibz.it/dis/teaching/DWDM/project2010/LogisticRegression.pdf (Section "Significance testing in ridge logistic regression")
	``` #Resolved

	// http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3228544/
	// http://www.aloki.hu/pdf/0402_171179.pdf (Equations 11 and 25)
	``` #Resolved

	<PackageReference Include="Newtonsoft.Json" Version="$(NewtonsoftJsonPackageVersion)" />
	<PackageReference Include="System.Reflection.Emit.Lightweight" Version="$(SystemReflectionEmitLightweightPackageVersion)" />
	<PackageReference Include="System.Threading.Tasks.Dataflow" Version="$(SystemThreadingTasksDataflowPackageVersion)" />
	<PackageReference Include="System.CodeDom" Version="$(SystemCodeDomPackageVersion)" />
	<PackageReference Include="System.Memory" Version="$(SystemMemoryVersion)" />
	<PackageReference Include="System.Collections.Immutable" Version="$(SystemCollectionsImmutableVersion)" />
	<PackageReference Include="System.ComponentModel.Composition" Version="$(SystemComponentModelCompositionVersion)" />

Adding training statistics for LR in the HAL learners package. #1392

Adding training statistics for LR in the HAL learners package. #1392

Uh oh!

Conversation

sfilipi commented Oct 26, 2018

Uh oh!

sfilipi Oct 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eerhardt Oct 26, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GalOshri commented Oct 26, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfilipi commented Oct 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomFinley left a comment

Choose a reason for hiding this comment

Uh oh!

klausmh commented Oct 31, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfilipi commented Nov 1, 2018

Uh oh!

klausmh commented Nov 1, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wschin Nov 5, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wschin Nov 5, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wschin Nov 5, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wschin Nov 5, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Nov 7, 2018

Choose a reason for hiding this comment

Uh oh!

wschin Nov 5, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomFinley left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Nov 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi Nov 8, 2018

Choose a reason for hiding this comment

Uh oh!

eerhardt Nov 8, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfilipi commented Nov 9, 2018

Uh oh!

sfilipi Nov 9, 2018

Choose a reason for hiding this comment

Uh oh!

eerhardt Nov 12, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eerhardt Nov 12, 2018 • edited by sfilipi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

sfilipi Oct 26, 2018 •

edited

Loading

eerhardt Oct 26, 2018 •

edited by sfilipi

Loading

GalOshri commented Oct 26, 2018 •

edited by sfilipi

Loading

sfilipi commented Oct 26, 2018 •

edited

Loading

klausmh commented Oct 31, 2018 •

edited by sfilipi

Loading

klausmh commented Nov 1, 2018 •

edited by sfilipi

Loading

wschin Nov 5, 2018 •

edited by sfilipi

Loading

wschin Nov 5, 2018 •

edited by sfilipi

Loading

wschin Nov 5, 2018 •

edited by sfilipi

Loading

wschin Nov 5, 2018 •

edited by sfilipi

Loading

wschin Nov 5, 2018 •

edited by sfilipi

Loading

TomFinley left a comment •

edited

Loading

sfilipi Nov 7, 2018 •

edited

Loading

eerhardt Nov 8, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt Nov 12, 2018 •

edited by sfilipi

Loading

eerhardt commented Nov 13, 2018 •

edited

Loading