Skip to content

Commit ebdf672

Browse files
author
Github Actions
committed
Ravin Kohli: [FIX] results management and visualisation with missing test data (#465)
1 parent 031db24 commit ebdf672

35 files changed

+211
-178
lines changed
Binary file not shown.
Binary file not shown.
Loading
Loading
Loading
Loading

development/_sources/examples/20_basics/example_image_classification.rst.txt

Lines changed: 14 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -33,22 +33,22 @@ Image Classification
3333
3434
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
3535
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to ../datasets/FashionMNIST/raw/train-images-idx3-ubyte.gz
36-
0%| | 0/26421880 [00:00<?, ?it/s] 0%| | 65536/26421880 [00:00<01:15, 351267.31it/s] 1%| | 163840/26421880 [00:00<00:43, 609414.50it/s] 2%|1 | 425984/26421880 [00:00<00:24, 1062279.85it/s] 5%|5 | 1343488/26421880 [00:00<00:07, 3392978.15it/s] 13%|#3 | 3473408/26421880 [00:00<00:03, 7060198.72it/s] 29%|##9 | 7733248/26421880 [00:00<00:01, 16103078.25it/s] 43%|####2 | 11304960/26421880 [00:00<00:00, 21320558.99it/s] 59%|#####9 | 15597568/26421880 [00:01<00:00, 22502874.39it/s] 75%|#######4 | 19791872/26421880 [00:01<00:00, 27264935.08it/s] 89%|########9 | 23592960/26421880 [00:01<00:00, 30051582.76it/s] 100%|##########| 26421880/26421880 [00:01<00:00, 18599895.80it/s]
36+
0%| | 0/26421880 [00:00<?, ?it/s] 0%| | 32768/26421880 [00:00<01:37, 269661.04it/s] 0%| | 65536/26421880 [00:00<01:38, 267501.14it/s] 0%| | 131072/26421880 [00:00<01:07, 388969.63it/s] 1%| | 229376/26421880 [00:00<00:47, 551190.13it/s] 2%|1 | 458752/26421880 [00:00<00:25, 1024519.07it/s] 4%|3 | 950272/26421880 [00:00<00:12, 2034589.74it/s] 7%|7 | 1867776/26421880 [00:00<00:06, 3806328.31it/s] 14%|#4 | 3768320/26421880 [00:00<00:03, 7511243.63it/s] 26%|##6 | 6881280/26421880 [00:01<00:01, 13055905.73it/s] 38%|###7 | 9994240/26421880 [00:01<00:00, 16843136.89it/s] 49%|####9 | 13008896/26421880 [00:01<00:00, 19076794.87it/s] 61%|###### | 16089088/26421880 [00:01<00:00, 20876086.01it/s] 73%|#######2 | 19169280/26421880 [00:01<00:00, 22121784.53it/s] 84%|########4 | 22249472/26421880 [00:01<00:00, 22991935.53it/s] 96%|#########5| 25329664/26421880 [00:01<00:00, 23587015.14it/s] 100%|##########| 26421880/26421880 [00:01<00:00, 14248915.59it/s]
3737
Extracting ../datasets/FashionMNIST/raw/train-images-idx3-ubyte.gz to ../datasets/FashionMNIST/raw
3838
3939
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
4040
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to ../datasets/FashionMNIST/raw/train-labels-idx1-ubyte.gz
41-
0%| | 0/29515 [00:00<?, ?it/s] 100%|##########| 29515/29515 [00:00<00:00, 326077.20it/s]
41+
0%| | 0/29515 [00:00<?, ?it/s] 100%|##########| 29515/29515 [00:00<00:00, 246986.62it/s] 100%|##########| 29515/29515 [00:00<00:00, 246193.85it/s]
4242
Extracting ../datasets/FashionMNIST/raw/train-labels-idx1-ubyte.gz to ../datasets/FashionMNIST/raw
4343

4444
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
4545
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to ../datasets/FashionMNIST/raw/t10k-images-idx3-ubyte.gz
46-
0%| | 0/4422102 [00:00<?, ?it/s] 1%|1 | 65536/4422102 [00:00<00:12, 357624.30it/s] 4%|4 | 196608/4422102 [00:00<00:05, 764127.24it/s] 11%|#1 | 491520/4422102 [00:00<00:03, 1253424.48it/s] 35%|###4 | 1540096/4422102 [00:00<00:00, 3979059.86it/s] 87%|########6 | 3833856/4422102 [00:00<00:00, 7916719.41it/s] 100%|##########| 4422102/4422102 [00:00<00:00, 6005980.87it/s]
46+
0%| | 0/4422102 [00:00<?, ?it/s] 1%| | 32768/4422102 [00:00<00:16, 271365.89it/s] 1%|1 | 65536/4422102 [00:00<00:16, 270266.93it/s] 3%|2 | 131072/4422102 [00:00<00:10, 392338.29it/s] 5%|5 | 229376/4422102 [00:00<00:07, 556637.94it/s] 10%|# | 458752/4422102 [00:00<00:03, 1036164.16it/s] 21%|## | 917504/4422102 [00:00<00:01, 1967160.72it/s] 41%|####1 | 1835008/4422102 [00:00<00:00, 3788204.48it/s] 83%|########2 | 3670016/4422102 [00:00<00:00, 7365292.41it/s] 100%|##########| 4422102/4422102 [00:00<00:00, 4526747.11it/s]
4747
Extracting ../datasets/FashionMNIST/raw/t10k-images-idx3-ubyte.gz to ../datasets/FashionMNIST/raw
4848

4949
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
5050
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to ../datasets/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz
51-
0%| | 0/5148 [00:00<?, ?it/s] 100%|##########| 5148/5148 [00:00<00:00, 30072809.18it/s]
51+
0%| | 0/5148 [00:00<?, ?it/s] 100%|##########| 5148/5148 [00:00<00:00, 31894057.60it/s]
5252
Extracting ../datasets/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to ../datasets/FashionMNIST/raw
5353

5454
Pipeline CS:
@@ -85,24 +85,23 @@ Image Classification
8585
Configuration(values={
8686
'image_augmenter:GaussianBlur:use_augmenter': False,
8787
'image_augmenter:GaussianNoise:use_augmenter': False,
88-
'image_augmenter:RandomAffine:rotate': 264,
89-
'image_augmenter:RandomAffine:scale_offset': 0.1997598286556691,
90-
'image_augmenter:RandomAffine:shear': 38,
91-
'image_augmenter:RandomAffine:translate_percent_offset': 0.21531300486072077,
88+
'image_augmenter:RandomAffine:rotate': 124,
89+
'image_augmenter:RandomAffine:scale_offset': 0.06255439515201121,
90+
'image_augmenter:RandomAffine:shear': 31,
91+
'image_augmenter:RandomAffine:translate_percent_offset': 0.17281138629598677,
9292
'image_augmenter:RandomAffine:use_augmenter': True,
93-
'image_augmenter:RandomCutout:p': 0.24443754548888208,
94-
'image_augmenter:RandomCutout:use_augmenter': True,
95-
'image_augmenter:Resize:use_augmenter': True,
96-
'image_augmenter:ZeroPadAndCrop:percent': 0.37202634450268485,
97-
'normalizer:__choice__': 'ImageNormalizer',
93+
'image_augmenter:RandomCutout:use_augmenter': False,
94+
'image_augmenter:Resize:use_augmenter': False,
95+
'image_augmenter:ZeroPadAndCrop:percent': 0.1857189312490744,
96+
'normalizer:__choice__': 'NoNormalizer',
9897
})
9998

10099
Fitting the pipeline...
101100
________________________________________
102101
ImageClassificationPipeline
103102
________________________________________
104103
0-) normalizer:
105-
ImageNormalizer
104+
NoNormalizer
106105

107106
1-) preprocessing:
108107
EarlyPreprocessing
@@ -174,7 +173,7 @@ Image Classification
174173
175174
.. rst-class:: sphx-glr-timing
176175

177-
**Total running time of the script:** ( 0 minutes 5.806 seconds)
176+
**Total running time of the script:** ( 0 minutes 6.707 seconds)
178177

179178

180179
.. _sphx_glr_download_examples_20_basics_example_image_classification.py:

development/_sources/examples/20_basics/example_tabular_classification.rst.txt

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -132,7 +132,7 @@ Search for an ensemble of machine learning algorithms
132132
.. code-block:: none
133133
134134
135-
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7f1b6277e6a0>
135+
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7fad73de7790>
136136
137137
138138
@@ -174,9 +174,9 @@ Print the final ensemble performance
174174
Optimisation Metric: accuracy
175175
Best validation score: 0.8713450292397661
176176
Number of target algorithm runs: 27
177-
Number of successful target algorithm runs: 25
177+
Number of successful target algorithm runs: 26
178178
Number of crashed target algorithm runs: 0
179-
Number of target algorithms that exceeded the time limit: 2
179+
Number of target algorithms that exceeded the time limit: 1
180180
Number of target algorithms that exceeded the memory limit: 0
181181
182182
@@ -186,7 +186,7 @@ Print the final ensemble performance
186186
187187
.. rst-class:: sphx-glr-timing
188188

189-
**Total running time of the script:** ( 5 minutes 31.873 seconds)
189+
**Total running time of the script:** ( 5 minutes 32.227 seconds)
190190

191191

192192
.. _sphx_glr_download_examples_20_basics_example_tabular_classification.py:

development/_sources/examples/20_basics/example_tabular_regression.rst.txt

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ Search for an ensemble of machine learning algorithms
123123
.. code-block:: none
124124
125125
126-
<autoPyTorch.api.tabular_regression.TabularRegressionTask object at 0x7f1ad2685730>
126+
<autoPyTorch.api.tabular_regression.TabularRegressionTask object at 0x7face7e10f40>
127127
128128
129129
@@ -163,7 +163,7 @@ Print the final ensemble performance
163163
| 2 | SimpleImputer,Variance Threshold,NoCoalescer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
164164
| 3 | None | LGBMLearner | 0.04 |
165165
autoPyTorch results:
166-
Dataset name: 63413bef-1a43-11ed-8830-056390bd6e17
166+
Dataset name: 70461479-1a43-11ed-883e-c7f5fe1042ee
167167
Optimisation Metric: r2
168168
Best validation score: 0.8670098636440993
169169
Number of target algorithm runs: 23
@@ -179,7 +179,7 @@ Print the final ensemble performance
179179
180180
.. rst-class:: sphx-glr-timing
181181

182-
**Total running time of the script:** ( 5 minutes 35.870 seconds)
182+
**Total running time of the script:** ( 5 minutes 37.001 seconds)
183183

184184

185185
.. _sphx_glr_download_examples_20_basics_example_tabular_regression.py:

development/_sources/examples/20_basics/example_time_series_forecasting.rst.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ Search for an ensemble of machine learning algorithms
150150
151151
.. rst-class:: sphx-glr-timing
152152

153-
**Total running time of the script:** ( 0 minutes 58.963 seconds)
153+
**Total running time of the script:** ( 1 minutes 8.766 seconds)
154154

155155

156156
.. _sphx_glr_download_examples_20_basics_example_time_series_forecasting.py:

development/_sources/examples/20_basics/sg_execution_times.rst.txt

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,14 @@
55

66
Computation times
77
=================
8-
**12:12.513** total execution time for **examples_20_basics** files:
8+
**12:24.700** total execution time for **examples_20_basics** files:
99

1010
+----------------------------------------------------------------------------------------------------------------+-----------+--------+
11-
| :ref:`sphx_glr_examples_20_basics_example_tabular_regression.py` (``example_tabular_regression.py``) | 05:35.870 | 0.0 MB |
11+
| :ref:`sphx_glr_examples_20_basics_example_tabular_regression.py` (``example_tabular_regression.py``) | 05:37.001 | 0.0 MB |
1212
+----------------------------------------------------------------------------------------------------------------+-----------+--------+
13-
| :ref:`sphx_glr_examples_20_basics_example_tabular_classification.py` (``example_tabular_classification.py``) | 05:31.873 | 0.0 MB |
13+
| :ref:`sphx_glr_examples_20_basics_example_tabular_classification.py` (``example_tabular_classification.py``) | 05:32.227 | 0.0 MB |
1414
+----------------------------------------------------------------------------------------------------------------+-----------+--------+
15-
| :ref:`sphx_glr_examples_20_basics_example_time_series_forecasting.py` (``example_time_series_forecasting.py``) | 00:58.963 | 0.0 MB |
15+
| :ref:`sphx_glr_examples_20_basics_example_time_series_forecasting.py` (``example_time_series_forecasting.py``) | 01:08.766 | 0.0 MB |
1616
+----------------------------------------------------------------------------------------------------------------+-----------+--------+
17-
| :ref:`sphx_glr_examples_20_basics_example_image_classification.py` (``example_image_classification.py``) | 00:05.806 | 0.0 MB |
17+
| :ref:`sphx_glr_examples_20_basics_example_image_classification.py` (``example_image_classification.py``) | 00:06.707 | 0.0 MB |
1818
+----------------------------------------------------------------------------------------------------------------+-----------+--------+

development/_sources/examples/40_advanced/example_custom_configuration_space.rst.txt

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,7 @@ Search for an ensemble of machine learning algorithms
161161
.. code-block:: none
162162
163163
164-
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7f1acad59730>
164+
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7face42bc940>
165165
166166
167167
@@ -205,7 +205,7 @@ Print the final ensemble performance
205205
| 9 | None | SVMLearner | 0.02 |
206206
| 10 | SimpleImputer,Variance Threshold,NoCoalescer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
207207
autoPyTorch results:
208-
Dataset name: 46889735-1a47-11ed-8830-056390bd6e17
208+
Dataset name: 701778c2-1a47-11ed-883e-c7f5fe1042ee
209209
Optimisation Metric: accuracy
210210
Best validation score: 0.8596491228070176
211211
Number of target algorithm runs: 18
@@ -268,7 +268,7 @@ Search for an ensemble of machine learning algorithms
268268
.. code-block:: none
269269
270270
271-
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7f1acab936a0>
271+
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7face7597520>
272272
273273
274274
@@ -308,13 +308,13 @@ Print the final ensemble performance
308308
| 6 | None | KNNLearner | 0.02 |
309309
| 7 | SimpleImputer,Variance Threshold,NoCoalescer,NoEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
310310
autoPyTorch results:
311-
Dataset name: b001e5c8-1a47-11ed-8830-056390bd6e17
311+
Dataset name: d86e2795-1a47-11ed-883e-c7f5fe1042ee
312312
Optimisation Metric: accuracy
313313
Best validation score: 0.8596491228070176
314-
Number of target algorithm runs: 20
314+
Number of target algorithm runs: 21
315315
Number of successful target algorithm runs: 15
316316
Number of crashed target algorithm runs: 5
317-
Number of target algorithms that exceeded the time limit: 0
317+
Number of target algorithms that exceeded the time limit: 1
318318
Number of target algorithms that exceeded the memory limit: 0
319319
320320
@@ -324,7 +324,7 @@ Print the final ensemble performance
324324
325325
.. rst-class:: sphx-glr-timing
326326

327-
**Total running time of the script:** ( 5 minutes 41.695 seconds)
327+
**Total running time of the script:** ( 5 minutes 46.156 seconds)
328328

329329

330330
.. _sphx_glr_download_examples_40_advanced_example_custom_configuration_space.py:

development/_sources/examples/40_advanced/example_parallel_n_jobs.rst.txt

Lines changed: 24 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,20 @@ with AutoPyTorch
3434

3535
.. code-block:: none
3636
37-
[ERROR] [2022-08-12 14:00:41,871:asyncio.events]
37+
[ERROR] [2022-08-12 14:01:49,186:asyncio.events]
38+
Traceback (most recent call last):
39+
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/utils.py", line 799, in wrapper
40+
return await func(*args, **kwargs)
41+
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/client.py", line 1246, in _reconnect
42+
await self._ensure_connected(timeout=timeout)
43+
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/client.py", line 1276, in _ensure_connected
44+
comm = await connect(
45+
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/comm/core.py", line 315, in connect
46+
await asyncio.sleep(backoff)
47+
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/asyncio/tasks.py", line 659, in sleep
48+
return await future
49+
asyncio.exceptions.CancelledError
50+
[ERROR] [2022-08-12 14:01:49,190:asyncio.events]
3851
Traceback (most recent call last):
3952
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/utils.py", line 799, in wrapper
4053
return await func(*args, **kwargs)
@@ -46,19 +59,19 @@ with AutoPyTorch
4659
await self._ensure_connected(timeout=timeout)
4760
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/client.py", line 1276, in _ensure_connected
4861
comm = await connect(
49-
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/comm/core.py", line 291, in connect
50-
comm = await asyncio.wait_for(
51-
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/asyncio/tasks.py", line 481, in wait_for
52-
await waiter
62+
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/distributed/comm/core.py", line 315, in connect
63+
await asyncio.sleep(backoff)
64+
File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/asyncio/tasks.py", line 659, in sleep
65+
return await future
5366
asyncio.exceptions.CancelledError
54-
{'accuracy': 0.9017341040462428}
67+
{'accuracy': 0.8497109826589595}
5568
autoPyTorch results:
56-
Dataset name: 5ddf95cd-1a46-11ed-8830-056390bd6e17
69+
Dataset name: 88611799-1a46-11ed-883e-c7f5fe1042ee
5770
Optimisation Metric: accuracy
5871
Best validation score: 0.8713450292397661
59-
Number of target algorithm runs: 44
60-
Number of successful target algorithm runs: 34
61-
Number of crashed target algorithm runs: 7
72+
Number of target algorithm runs: 46
73+
Number of successful target algorithm runs: 37
74+
Number of crashed target algorithm runs: 6
6275
Number of target algorithms that exceeded the time limit: 3
6376
Number of target algorithms that exceeded the memory limit: 0
6477
@@ -136,7 +149,7 @@ with AutoPyTorch
136149
137150
.. rst-class:: sphx-glr-timing
138151

139-
**Total running time of the script:** ( 5 minutes 46.611 seconds)
152+
**Total running time of the script:** ( 5 minutes 44.538 seconds)
140153

141154

142155
.. _sphx_glr_download_examples_40_advanced_example_parallel_n_jobs.py:

development/_sources/examples/40_advanced/example_pass_feature_types.rst.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ Search for an ensemble of machine learning algorithms
151151
.. code-block:: none
152152
153153
154-
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7f1ad16c85b0>
154+
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7face7b0b220>
155155
156156
157157
@@ -203,7 +203,7 @@ Print the final ensemble performance
203203
204204
.. rst-class:: sphx-glr-timing
205205

206-
**Total running time of the script:** ( 1 minutes 49.275 seconds)
206+
**Total running time of the script:** ( 1 minutes 50.152 seconds)
207207

208208

209209
.. _sphx_glr_download_examples_40_advanced_example_pass_feature_types.py:

development/_sources/examples/40_advanced/example_plot_over_time.rst.txt

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -87,9 +87,9 @@ Task Definition
8787

8888
.. code-block:: none
8989
90-
[0 1 0 0 0 0 1 0 1 0 0 0 1 0 1 0 1 0 1 0 1 1 1 1 0 1 0 1 0 1 1 1 0 1 0 1 0
91-
1 0 0 1 1 0 0 0 1 0 0 1 1 0 1 0 0 0 0 0 0 0 1 0 1 1 1 1 1 1 0 1 1 0 0 1 1
92-
0 1 0 0 0 0 0 1 1 1 0 1 1 1 0 0 0 0 1 0 0 1 0 1 1 1]
90+
[1 1 1 0 0 1 1 0 1 0 0 1 1 0 0 0 1 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 1 1 0 0
91+
0 1 0 0 1 0 0 0 1 0 1 1 1 1 0 1 0 1 1 1 1 1 0 1 0 1 0 1 0 0 1 1 1 1 0 1 1
92+
0 1 1 1 1 1 1 1 1 0 1 0 1 0 1 0 0 0 0 0 1 1 1 0 1 1]
9393
9494
9595
@@ -117,7 +117,7 @@ API Instantiation and Searching
117117
.. code-block:: none
118118
119119
120-
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7f1acb3fdf40>
120+
<autoPyTorch.api.tabular_classification.TabularClassificationTask object at 0x7fad65e3bb20>
121121
122122
123123
@@ -182,7 +182,7 @@ _, ax = plt.subplots() <=== You can feed it to post-process the figure.
182182

183183
.. rst-class:: sphx-glr-timing
184184

185-
**Total running time of the script:** ( 2 minutes 11.558 seconds)
185+
**Total running time of the script:** ( 2 minutes 22.976 seconds)
186186

187187

188188
.. _sphx_glr_download_examples_40_advanced_example_plot_over_time.py:

0 commit comments

Comments
 (0)