TM030 Check Results API

Check Results API

Goal: define and implement a (JavaScript) interface that can be used from both pyret-lang and code.pyret.org that easily transforms the monolithic, black-boxed "run result" into a relatively-flat plain object/record which gives a detailed account of either errors or all check results.

Potential Use Cases

• Automated grading, e.g. for Brown's CS019 and CS173, tends to be an iterated exercise in complexity which requires some TA have understanding of Pyret internals. An API like this, supported from pyret-lang, would make that much easier. NOTE: We aren't discussing being able to override the my-gdrive and/or shared-gdrive imports in this document, but that would also be nice.

• code.pyret.org could use this to save the headache of maintaining separate code that must do runtime/stack management in order to dive into Pyret objects to extract relevant data

Example: Successfully Ran

We would like to be able to run the following program:

check "first block":
  empty.first raises "not-found"
  empty.first raises "foo"
  [list: 1,2,3,4,5].first is 1
  [list: 2,4,6,8].first is 1
end

check "second block":
  [list: 1].first raises "not-found"
end

and get data that looks roughly like this:

{
    "timestamp": 1481128853291,
    "stats": {
            "full": {
                "bounces":12,
                "tos":15,
                "time":3195.9799999999996
            },
            "run": {
                "bounces":4,
                "tos":4,
                "time":51.36999999999898
            }
        }
    },
    "error": null,
    "checks": {
        "testsPassed": 2,
        "testsTotal": 5,
        "numCheckBlocks": 2,
        "checkBlocks": [
            {
                "name": "Value tests",
                "error": null,
                "testsPassedInBlock": 2,
                "testsRanInBlock": 4,
                "testsTotalInBlock": 4,
                "tests": [
                    {
                        "passed": true,
                        "result": "success",
                        "reason": null,
                        "loc": {
                            "source": "definitions://",
                            "line": 2
                        }
                    },
                    {
                        "passed": false,
                        "result": "failure-wrong-exn",
                        "reason": "Got unexpected exception\n\"not-found\"\nwhen expecting\n\"foo\"",
                        "loc": {
                            "source": "definitions://",
                            "line": 3
                        }
                    },
                    {
                        "isSuccess": true,
                        "result": "success",
                        "reason": null,
                        "loc": {
                            "source": "definitions://",
                            "line": 4
                        }
                    },
                    {
                        "isSuccess": false,
                        "result": "failure-not-equal",
                        "reason": "Values not equal:\n2\n1",
                        "loc": {
                            "source": "definitions://",
                            "line": 5
                        }
                    }
                ]
            },
            {
                "name": "Operator tests",
                "error": null,
                "testsPassedInBlock": 0,
                "testsRanInBlock": 1,
                "testsTotalInBlock": 1,
                "tests": [
                    {
                        "isSuccess": false,
                        "result": "failure-no-exn",
                        "reason": "No exception raised, expected\n\"not-found\"",
                        "loc": {
                            "source": "definitions://",
                            "line": 9
                        }
                    },
                ]
            }
        ]
    }
}