tutorial - getting started doc - smoke test job fail with 502 Server Error: Bad Gateway

[CloudBreadPaPa]

Subject : tutorial - getting started doc - smoke test job fail with 502 Server Error: Bad Gateway

Symptom : pipeline fail with 502 Server Error: Bad Gateway

Detail error message :

...
502 Server Error: Bad Gateway for url: http://<ACI-HOST>.centralus.azurecontainer.io/score
Retrying...
502 Server Error: Bad Gateway for url: http://<ACI-HOST>.centralus.azurecontainer.io/score
Retrying...
url : http://<ACI-HOST>.centralus.azurecontainer.io/score
input : {'data': [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]]}
headers : {'traceparent': '00-722be4574da391d0a6894d190ddc4a25-dc4ead82e44ac27a-00'}
Traceback (most recent call last):
  File "ml_service/util/smoke_test_scoring_service.py", line 97, in <module>
    main()
  File "ml_service/util/smoke_test_scoring_service.py", line 88, in main
    output = call_web_service(e, args.type, args.service)
  File "ml_service/util/smoke_test_scoring_service.py", line 35, in call_web_service
    output = call_web_app(service.scoring_uri, headers)
  File "ml_service/util/smoke_test_scoring_service.py", line 59, in call_web_app
    raise e
  File "ml_service/util/smoke_test_scoring_service.py", line 55, in call_web_app
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: http://<ACI-HOST>.centralus.azurecontainer.io/score
##[error]Script failed with error: Error: The process '/bin/bash' failed with exit code 1
/usr/local/bin/az account clear
Finishing: Smoke test

Approach :
"/ml_service/util/smoke_test_scoring_service.py" might issue on it.
Built test env for this smoke test score and tested.
The detail error message was "502 Server Error: Bad Gateway error" with below error message

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 11 is different from 10)

• model file need to input (11, 1) shape not (10, 1) shape.
• used this datafile to train model : https://github.com/microsoft/MLOpsPython/blob/master/data/diabetes.csv

Resolution :
Replaced "/ml_service/util/smoke_test_scoring_service.py" file input data shape

input = {"data": [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
                  [11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1]]}

Replace "/diabetes_regression/scoring/score.py" file input input data

input_sample = numpy.array([
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
    [11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1]])

and

test_row = '{"data":[[1,2,3,4,5,6,7,8,9,10,11],[11,10,9,8,7,6,5,4,3,2,1]]}'

• removed error pipeline Azure DevOps pipeline
• removed deployment and model on Azure Machine Learning Workspace
• recreated pipeline

The smoke test passed and pipeline works.

[romanlytvyn]

Looks like this fix does not address the core issue.
The input dataset contains 11 columns: 10 features and 1 label column.
There is an issue in the model training script, as output labels are included in the input features list.
If we exclude them, the scoring behaves as expected, accepting an array of 10 features.

Fixed by #198

[stevehaigh]

Thanks @romanlytvyn , I just came to create a PR for this, now I don't have to :)