Added categorical value support for PredictedLabel for the Image Classification Transfer Learning example. Addresses Issue #4169 #4228

mstfbl · 2019-09-18T20:36:52Z

Attempt on Issue #4169

Added the necessary estimator pipeline for a KeyDataViewType string/value binding
Changed the Output Column types of the ImageClassificationEstimator and the Mapper for future compatibility with KeyType composite values
Added test cases to test features added above

Attempt on Issue 4169 - Added the necessary estimator pipeline for a KeyDataViewType string/value binding - Changed the Output Column types of the ImageClassificationEstimator and the Mapper for future compatibility with KeyType composite values

dnfclas · 2019-09-18T20:37:07Z

All CLA requirements met. #Resolved

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs

codemzs · 2019-09-18T20:56:58Z

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs

                EvaluateModel(mlContext, testDataset, loadedModel);

                VBuffer<ReadOnlyMemory<char>> keys = default;
                loadedModel.GetOutputSchema(schema)["Label"].GetKeyValues(ref keys);


shouldn't need to do all this if you convert predicted label column from Key to Value. It will output string type that will contain class names that corresponds to key indices. #Resolved

codemzs · 2019-09-18T20:57:21Z

        public UInt32 PredictedLabel;

this needs to be string. #Resolved

Refers to: docs/samples/Microsoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs:308 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs

codemzs · 2019-09-18T20:57:53Z

            $"Predicted Label : {originalLabels[index]}");

prediction.PredictedLabel. #Resolved

Refers to: docs/samples/Microsoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs:148 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

codemzs

🕐

yaeldekel · 2019-09-19T15:05:34Z

                : SchemaShape.Column.VectorKind.VariableVector, _outputTypes[i].GetItemType(), false);

This is wrong. Given how _outputTypes is initialized in line 1169, the outcome of the condition here is always false: _outputType[0] is a variable length vector, and _outputTypes[1] is a scalar. This would make the result always be of kind SchemaShape.Column.VectorKind.VariableVector, which is not the case. #Resolved

Refers to: src/Microsoft.ML.Dnn/ImageClassificationTransform.cs:1203 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

yaeldekel · 2019-09-19T15:09:18Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

            _dnnModel = dnnModel;
            _tfInputTypes = new[] { TF_DataType.TF_STRING };
-            _outputTypes = new[] { new VectorDataViewType(NumberDataViewType.Single), NumberDataViewType.UInt32.GetItemType() };
+            _outputTypes = new DataViewType[] { new VectorDataViewType(NumberDataViewType.Single), new KeyDataViewType(typeof(uint), 5) };


_outputTypes [](start = 12, length = 12)

_outputTypes doesn't need to be an array of DataViewType. All the estimator needs to know is whether the length of the score vector is variable or not (seems to me that it is typically not variable, or at least this information can be inferred from the DnnModel).
If you get rid of this field, you will not have to guess the size of the key (which is also not needed by the estimator). #Resolved

This is a good catch, thank you! Quick question, the Image Classification example ResnetV2101TransferLearningTrainTestSplit.cs runs perfectly well when I don't further define _outputTypes, i,e, delete line 69. Does this also mean than the estimator isn't using this field? #Resolved

Actually, I've found that the DataViewType field is needed for the pipeline in ResnetV2101TransferLearningTrainTestSplit.cs to fit properly. #Resolved

Can you elaborate? The only place _outputTypes is used is in GetOutputSchema (and it is actually being used incorrectly there - see my comments there). What is not working properly in the example you mentioned?

In reply to: 326329163 [](ancestors = 326329163)

It was giving a different error when I removed _outputTypes, but as I now see that was not the bottleneck problem I was having while implementing this KeyType solution. #Resolved

The field _outputTypes can be deleted. The only place where it is used is in line 1281 where you have _outputTypes[0].GetItemType(). Instead of that you can write NumberDataViewType.Single, and then the field is not needed.

In reply to: 326229209 [](ancestors = 326229209)

I missed this, thank you for the catch! #Resolved

yaeldekel · 2019-09-19T15:17:25Z

        for (var i = 0; i < _options.OutputColumns.Length; i++)

How does this work?
It seems like there is an assumption here that the output columns array is always of length 2, and always contains the score column and the predicted label column, in that order. If this is true, then this argument needs to be replaced with two string arguments for the score column name and the predicted label column name. #Resolved

Refers to: src/Microsoft.ML.Dnn/ImageClassificationTransform.cs:1199 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

yaeldekel · 2019-09-19T15:19:14Z

                : SchemaShape.Column.VectorKind.VariableVector, _outputTypes[i].GetItemType(), false);

This needs to output exactly two SchemaShape.Columns: The first with the score column name, of kind VectorKind.Vector, and the other with the predicted label column name, of kind VectorKind.Scalar, and with isKey set to true.

In reply to: 533174369 [](ancestors = 533174369)

Refers to: src/Microsoft.ML.Dnn/ImageClassificationTransform.cs:1203 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

yaeldekel · 2019-09-19T15:20:27Z

        _labelColumnName = labelColumnName;

This is not used anywhere but at serialization time. Why does this need to be serialized? #Resolved

Refers to: src/Microsoft.ML.Dnn/ImageClassificationTransform.cs:667 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

yaeldekel · 2019-09-19T15:26:05Z

        bool? addBatchDimensionInput, int batchSize, string labelColumnName, string finalModelPrefix, Architecture arch,

This is where we have both the label column name and the training data schema, so this is where we need to extract the key values (if they exist). They will need to be serialized with the transformer, because at load time we don't have access to the training data. #Resolved

Refers to: src/Microsoft.ML.Dnn/ImageClassificationTransform.cs:652 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

yaeldekel · 2019-09-19T15:26:59Z

        bool? addBatchDimensionInput, int batchSize, string labelColumnName, string finalModelPrefix, Architecture arch,

I think this argument is not used here. #Resolved

Refers to: src/Microsoft.ML.Dnn/ImageClassificationTransform.cs:652 in 8d07401. [](commit_id = 8d07401, deletion_comment = False)

yaeldekel

🕐

Here, I isolated the issue of the annotations which carry the KeyValues not copying over to the SchemaShapes in the SchemaShape Create method in IEstimator.cs. In this method, the KeyValues are not being recognized as KeyType, and as a result the annotations are not being copied over.

mstfbl · 2019-09-20T07:24:39Z

With my latest commit, I have isolated the issue to the annotations of KeyType not copying over to the SchemaShapes that are being created in line 194 of IEstimator.cs and subsequently in line 61 of EstimatorChains.cs, which are part of the Fitting process in line 81 of ResnetV2101TransferLearningTrainTestSplit.cs . If we can incorporate a way of preserving the original KeyType annotations after the process, we can achieve a KeyType implementation when the Prediction Engine is run in the TrySinglePrediction method of ResnetV2101TransferLearningTrainTestSplit.cs. #Resolved

yaeldekel · 2019-09-20T12:54:07Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

                var info = new DataViewSchema.DetachedColumn[_parent._outputs.Length];
                info[0] = new DataViewSchema.DetachedColumn(_parent._outputs[0], new VectorDataViewType(NumberDataViewType.Single, _parent._classCount), null);
-                info[1] = new DataViewSchema.DetachedColumn(_parent._outputs[1], NumberDataViewType.UInt32, null);
+                info[1] = new DataViewSchema.DetachedColumn(_parent._outputs[1], new KeyDataViewType(typeof(uint), _parent._classCount), ((DataViewSchema.Column)InputSchema.GetColumnOrNull(_parent._labelColumnName)).Annotations);


_labelColumnName [](start = 197, length = 16)

We cannot rely on the _labelColumnName column in the transformer, since it may not be present in InputSchema.
The way this should work is by getting the key values at the time we have the training data (which is required to contain the label column), which is in the constructor of the transformer. The key values should be serialized as part of saving the transformer, so that they are available even when the model is loaded from disk.
You should also be able to remove the _labelColumnName field from the transformer, it is not needed. #Resolved

src/Microsoft.ML.Data/DataLoadSave/EstimatorChain.cs

yaeldekel · 2019-09-20T13:07:28Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

                var info = new DataViewSchema.DetachedColumn[_parent._outputs.Length];
                info[0] = new DataViewSchema.DetachedColumn(_parent._outputs[0], new VectorDataViewType(NumberDataViewType.Single, _parent._classCount), null);
-                info[1] = new DataViewSchema.DetachedColumn(_parent._outputs[1], NumberDataViewType.UInt32, null);
+                info[1] = new DataViewSchema.DetachedColumn(_parent._outputs[1], new KeyDataViewType(typeof(uint), _parent._classCount), ((DataViewSchema.Column)InputSchema.GetColumnOrNull(_parent._labelColumnName)).Annotations);


_outputs [](start = 68, length = 8)

Please replace this field with two fields containing the score column name and the predicted label column name, and do the same for the argument in the Options class.
It is implicitly assumed throughout this code that there are exactly two output columns, the first one being the score column and the second one being the predicted label column - for example, this method (GetOutputColumnsCore), GetOutputSchema in the estimator, GetOutputColumnsCore in the mapper. #Resolved

I just noticed that there is the same problem with the input column names - on one hand the argument in the Options class is an array, and in other place it is assumed that it contains exactly one element which is the features column.

In reply to: 326619105 [](ancestors = 326619105)

yaeldekel · 2019-09-22T06:25:16Z

SchemaShape is a class that holds some partial information about the schema that the estimators can provide. Estimators are not able to give the exact schema, since they do not see any data until Fit is called. For this reason, they are only able to give information such as:

The column that is created is a fixed length vector (what is the length? the estimator doesn't know, it can know only when it sees the data).
The column that is created is a key type, and it has an annotation of key values that is a fixed length vector of text (what is the length of the key values vector and what are the actual key values? There is no way to know until we see the data).

Each estimator is in charge of creating the SchemaShape of its output. The EstimatorChain cannot know that it needs to copy annotations from a different column (from the label column to the predicted label column), the ImageClassificationEstimator is the component that has to do that, in its implementation of GetOutputSchema. Same with the transformer - the ImageClassificationTransformer needs to store the key values from the label column when it sees the training data, and pass them to the predicted label column.

In reply to: 533441383 [](ancestors = 533441383)

yaeldekel · 2019-09-24T08:10:53Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

            string[] inputColumnNames, string modelLocation,
            bool? addBatchDimensionInput, int batchSize, string labelColumnName, string finalModelPrefix, Architecture arch,
-            string scoreColumnName, string predictedLabelColumnName, float learningRate, DataViewSchema inputSchema, int? classCount = null, bool loadModel = false,
+            string scoreColumnName, string predictedLabelColumnName, float learningRate, DataViewSchema inputSchema, int? classCount = null, string[] keyValueAnnotations = null, bool loadModel = false,


keyValueAnnotations [](start = 150, length = 19)

The transformer can be created in one of two ways: Either by calling Fit on the estimator, or by deserializing from file. In the second case, the key values come from reading them from the model as you have written. In the first case, you need to get them from inputSchema by getting the key value annotations of the training label column. #Resolved

Notice that it is possible that the label column does not contain key values annotations, which means that the PredictedLabel column will not have them either. You have to account for that at serialization/deserialization time (for example, by serialiazing the number of key value strings, and only then serializing the strings them selves).

In reply to: 327481050 [](ancestors = 327481050)

yaeldekel · 2019-09-24T08:14:52Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

                _classCount = labelCount == 1 ? 2 : (int)labelCount;
+
+                //Temporary string array for key values, will replace soon
+                _keyValueAnnotations = new string[] { "sunflowers", "roses", "dandelion", "daisy", "tulips"};


_keyValueAnnotations [](start = 16, length = 20)

It seems that the loadModel argument specifies whether we are training the model or loading it from a file, inside the if in line 718 is where you should extract the key values from the input schema. #Resolved

yaeldekel · 2019-09-24T08:15:49Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

@@ -684,6 +692,9 @@ internal ImageClassificationTransformer(IHostEnvironment env, Session session, s
                    throw Host.ExceptSchemaMismatch(nameof(inputSchema), "label", (string)labelColumn.Name, "Key", (string)labelType.ToString());

                _classCount = labelCount == 1 ? 2 : (int)labelCount;


labelCount == 1 ? 2 : (int)labelCount [](start = 30, length = 37)

Why?
/cc @codemzs
#Resolved

To account for binary classification case where in the train set all you have seen is just one class :-).

In reply to: 327483161 [](ancestors = 327483161)

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

Implemented changes

codemzs · 2019-10-02T21:27:14Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+            Assert.Equal((int)labelCountFirst, predictionFirst.Score.Length);
+            Assert.Equal((int)labelCountLast, predictionLast.Score.Length);
+            Assert.True(Array.IndexOf(labels, predictionFirst.PredictedLabel) > -1);
+            Assert.True(Array.IndexOf(labels, predictionLast.PredictedLabel) > -1);


Can you please add an assert that predictionFirst and predictionLast have the same value? and also put an assert on the value? (it should remain the same since your seed is fixed) #Resolved

I'm sorry I meant can you please assert on the value for predictionFirst.PredictedLabel and predictionLast.PredictedLabel ... instead of just checking its greater than -1

In reply to: 330779542 [](ancestors = 330779542)

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

codemzs · 2019-10-02T21:32:03Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

+            Host.AssertNonEmpty(_keyValueAnnotations);
+            Host.Assert(_keyValueAnnotations.Length == _classCount);
+            for (int j = 0; j < _classCount; j++)
+                ctx.SaveNonEmptyString(_keyValueAnnotations[j]);


Is it really guaranteed _keyValueAnnotations is not empty? refer to below lines of code that you have written in the constructor.

VBuffer<ReadOnlyMemory> keysVBuffer = default;
if (inputSchema[labelColumnName].HasKeyValues())
{
inputSchema[labelColumnName].GetKeyValues(ref keysVBuffer);
_keyValueAnnotations = keysVBuffer.DenseValues().ToArray();
} #Resolved

With your suggested changes below, it is now guaranteed that _keyValueAnnotations will not be empty. #Resolved

codemzs · 2019-10-02T21:34:32Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

+                {
+                    inputSchema[labelColumnName].GetKeyValues(ref keysVBuffer);
+                    _keyValueAnnotations = keysVBuffer.DenseValues().ToArray();
+                }


If it does not have key values then just add numeric values (0 - classCount -1) as string to have something in there #Resolved

codemzs · 2019-10-02T22:08:00Z

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs

+                else
+                {
+                    _keyValueAnnotations = Enumerable.Range(0, _classCount).ToArray().Select(x => x.ToString().AsMemory()).ToArray();
+                }


Enumerable.Range(0, _classCount).Select(x => x.ToString().AsMemory()).ToArray(); #Resolved

codemzs · 2019-10-02T23:23:18Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+            for (int j = 0; j < predictionFirst.Score.Length; j++)
+            {
+                predictionFirst.Score[j] = predictionLast.Score[j];
+            }


I'm sorry what is this for?

My apologies, it's a mistake.

codemzs

codemzs · 2019-10-03T01:12:34Z

@mstfbl How come the tests are now failing?

codemzs · 2019-10-03T01:14:13Z

It seems your tests are failing because of the below error:

tarting test: Microsoft.ML.Scenarios.TensorFlowScenariosTests.TensorFlowSentimentClassificationTest
[xUnit.net 00:09:54.21] Microsoft.ML.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification [FAIL]
X Microsoft.ML.Scenarios.TensorFlowScenariosTests.TensorFlowImageClassification [1m 14s]
Error Message:
Assert.Equal() Failure
↓ (pos 0)
Expected: dandelion
Actual: sunflowers
↑ (pos 0)

Can you please make the tests deterministic by always passing the same image?

codemzs · 2019-10-03T01:18:52Z

test/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/TensorflowTests.cs

+            {
+                var dir = new DirectoryInfo(directories[j]);
+                labels[j] = dir.Name;
+            }


this is problematic, you cannot assume the ValueToKey transform will see the directories as labels in this order. Instead get keys from the IDataview like we were doing in the sample before

VBuffer<ReadOnlyMemory> keys = default;
loadedModel.GetOutputSchema(schema)["Label"].GetKeyValues(ref keys);

codemzs

your tests are failing/giving inaccurate results. we want to be sure you are setting annotations correctly before we merge this,.

mstfbl · 2019-10-03T06:04:05Z

I just manually tested for predicting two known images for their labels using the master, unmerged code. I tested predicting an image of a daisy and an image of a rose. The predictions here are correct, which means the error is in my additions pertaining to my annotations, and not in the trained model. I have attached a screenshot of this below.

@yaeldekel @eerhardt We've determined that the trained model is performing correctly without the current additions of categorical values to PredictedLabel. Something is going on with my annotation additions, and I can't find the bug. Would you guys be able to help in diagnosing where the bug is in my additions with fresh new eyes?

codemzs · 2019-10-03T06:31:22Z

@mstfbl Your bug is in predictedLabel value in output cache. In KeyValue type Keys start from 1 but in Tensorflow predictedLabels start from 0. In file: ImageClassificationTransform.cs after Line 915 outputTensor[1].ToScalar(ref _predictedLabel); add _predictedLabel += 1;

codemzs · 2019-10-03T06:52:28Z

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs

                    learningRate: 0.01f,
                    metricsCallback: (metrics) => Console.WriteLine(metrics),
+                    // reuseTrainSetBottleneckCachedValues: true,
+                    // reuseValidationSetBottleneckCachedValues: true,


Please remove this

codemzs · 2019-10-03T06:52:45Z

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs

-                    arch: ImageClassificationEstimator.Architecture.ResnetV2101, 
+                    // Uncomment reuseTrainSetBottleneckCachedValues and
+                    // reuseValidationSetBottleneckCachedValues to reuse trained model
+                    // for faster debugging.


Please remove this

codemzs

Attempt on Issue 4169

8d07401

Attempt on Issue 4169 - Added the necessary estimator pipeline for a KeyDataViewType string/value binding - Changed the Output Column types of the ImageClassificationEstimator and the Mapper for future compatibility with KeyType composite values

codemzs reviewed Sep 18, 2019

View reviewed changes

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs Outdated Show resolved Hide resolved

codemzs reviewed Sep 18, 2019

View reviewed changes

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs Outdated Show resolved Hide resolved

codemzs reviewed Sep 18, 2019

View reviewed changes

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs Outdated Show resolved Hide resolved

codemzs reviewed Sep 18, 2019

View reviewed changes

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs Outdated Show resolved Hide resolved

codemzs reviewed Sep 18, 2019

View reviewed changes

...icrosoft.ML.Samples/Dynamic/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs Outdated Show resolved Hide resolved

codemzs requested changes Sep 18, 2019

View reviewed changes

yaeldekel reviewed Sep 19, 2019

View reviewed changes

yaeldekel previously requested changes Sep 19, 2019

View reviewed changes

yaeldekel reviewed Sep 20, 2019

View reviewed changes

src/Microsoft.ML.Data/DataLoadSave/EstimatorChain.cs Outdated Show resolved Hide resolved

yaeldekel reviewed Sep 20, 2019

View reviewed changes

Temporary change for inquiry

7b239b0

yaeldekel reviewed Sep 24, 2019

View reviewed changes

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs Outdated Show resolved Hide resolved

codemzs reviewed Oct 2, 2019

View reviewed changes

src/Microsoft.ML.Dnn/ImageClassificationTransform.cs Outdated Show resolved Hide resolved

codemzs reviewed Oct 2, 2019

View reviewed changes

mstfbl added 2 commits October 2, 2019 14:55

Update ImageClassificationTransform.cs

b6cfeda

Added test case to check the matching of predicted labels

cca4fb8

codemzs reviewed Oct 2, 2019

View reviewed changes

Update ImageClassificationTransform.cs

1c4c5dc

codemzs reviewed Oct 2, 2019

View reviewed changes

mstfbl added 2 commits October 2, 2019 16:27

Update TensorflowTests.cs

2301a96

Update TensorflowTests.cs

26e2ae1

codemzs approved these changes Oct 2, 2019

View reviewed changes

codemzs reviewed Oct 3, 2019

View reviewed changes

Update TensorflowTests.cs

94c0423

codemzs requested changes Oct 3, 2019

View reviewed changes

mstfbl requested a review from yaeldekel October 3, 2019 06:04

mstfbl added 2 commits October 2, 2019 23:42

Fixed test case and off-by-one predictedLabel error

109879b

Merge remote-tracking branch 'upstream/master' into Issue_4169

6ff4c64

codemzs reviewed Oct 3, 2019

View reviewed changes

mstfbl added 2 commits October 2, 2019 23:55

Removed comments

15c4654

Replacing Path.Join with Path.Combine due to build error with Path.Join

0573ef3

codemzs approved these changes Oct 3, 2019

View reviewed changes

codemzs merged commit 4618fca into dotnet:master Oct 3, 2019

ghost locked as resolved and limited conversation to collaborators Mar 20, 2022

		@@ -684,6 +692,9 @@ internal ImageClassificationTransformer(IHostEnvironment env, Session session, s
		throw Host.ExceptSchemaMismatch(nameof(inputSchema), "label", (string)labelColumn.Name, "Key", (string)labelType.ToString());

		_classCount = labelCount == 1 ? 2 : (int)labelCount;

Added categorical value support for PredictedLabel for the Image Classification Transfer Learning example. Addresses Issue #4169 #4228

Added categorical value support for PredictedLabel for the Image Classification Transfer Learning example. Addresses Issue #4169 #4228

Uh oh!

Conversation

mstfbl commented Sep 18, 2019 • edited by codemzs Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dnfclas commented Sep 18, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codemzs Sep 18, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codemzs commented Sep 18, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codemzs commented Sep 18, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codemzs left a comment

Choose a reason for hiding this comment

Uh oh!

yaeldekel commented Sep 19, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaeldekel Sep 19, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstfbl Sep 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstfbl Sep 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstfbl Sep 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mstfbl Oct 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaeldekel commented Sep 19, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaeldekel commented Sep 19, 2019

Uh oh!

yaeldekel commented Sep 19, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaeldekel commented Sep 19, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaeldekel commented Sep 19, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaeldekel left a comment

Choose a reason for hiding this comment

Uh oh!

mstfbl commented Sep 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaeldekel Sep 20, 2019 • edited by mstfbl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mstfbl commented Sep 18, 2019 •

edited by codemzs

Loading

dnfclas commented Sep 18, 2019 •

edited by mstfbl

Loading

codemzs Sep 18, 2019 •

edited by mstfbl

Loading

codemzs commented Sep 18, 2019 •

edited by mstfbl

Loading

codemzs commented Sep 18, 2019 •

edited by mstfbl

Loading

yaeldekel commented Sep 19, 2019 •

edited by mstfbl

Loading

yaeldekel Sep 19, 2019 •

edited by mstfbl

Loading

mstfbl Sep 19, 2019 •

edited

Loading

mstfbl Sep 19, 2019 •

edited

Loading

mstfbl Sep 20, 2019 •

edited

Loading

mstfbl Oct 2, 2019 •

edited

Loading

yaeldekel commented Sep 19, 2019 •

edited by mstfbl

Loading

yaeldekel commented Sep 19, 2019 •

edited by mstfbl

Loading

yaeldekel commented Sep 19, 2019 •

edited by mstfbl

Loading

yaeldekel commented Sep 19, 2019 •

edited by mstfbl

Loading

mstfbl commented Sep 20, 2019 •

edited

Loading

yaeldekel Sep 20, 2019 •

edited by mstfbl

Loading

yaeldekel Sep 20, 2019 •

edited by mstfbl

Loading

yaeldekel Sep 24, 2019 •

edited by mstfbl

Loading

yaeldekel Sep 24, 2019 •

edited by mstfbl

Loading

yaeldekel Sep 24, 2019 •

edited by mstfbl

Loading

codemzs Oct 2, 2019 •

edited by mstfbl

Loading

codemzs Oct 2, 2019 •

edited by mstfbl

Loading

mstfbl Oct 2, 2019 •

edited

Loading

codemzs Oct 2, 2019 •

edited by mstfbl

Loading

codemzs Oct 2, 2019 •

edited by mstfbl

Loading