Skip to content

Evaluation Manifest

Jump to bottom
diamond_dragon edited this page Feb 13, 2022 · 4 revisions

Evaluation Manifest

Overview

One of features available in CodeReview.Evaluator is ability to execute queries against SQLite database and persist execution results in JSON. In order to do this evaluate command must be used.

evaluate command accepts two parameters as input:

  • Path to SQLite database. There are no restrictions on database which can be passed as parameter. It should be valid SQLite database.
  • Path to evaluation manifest. This is yaml document which contains queries to execute against SQLite database.
  • Path output file. Output file is json document containing execution results.

Data specified in SQLite database can be used to SQL query execution. Execution result is persisted in form of JSON document.

Queries vs Requests

Evaluation manifest contains two top level sections: queries and requests.

  • queries section defines list of queries which are later can be referred by requests using queryRef property. To some extend this section is collection of named queries. This solution allows to avoid query duplication when same query is used by multiple requests with different parameters.

  • requests is named collection requests which are executed during evaluation. Each require may have inline query defined by query property or may refer to existing query via queryRef property.

Request Types

There several request types which are supported by evalutor:

Parameter Type Is Default Description
Scalar True Request produces single value as result.
Object False Result is single object. Each column of result becomes a property of JSON document.
Collection False Request returns list of objects. Each column of result becomes a property of object. All objects are mapped to JSON array.
NoResult False Requests of this type might be used when non-select queries need to be executed. For example it might be DDL or DDM queries (DELETE, UPDATE, CREATE and etc.). These queries might be useful when data stored in SQLite database requires additional processing before select statements are executed. It's recommended to put requests of this type in the begining of requests collection. As result these requests are executed before select statements.

Scalar Type Example

Manifest:

requests:
  totalIssueCount:
    query: SELECT COUNT(*) FROM Issues
    type: Scalar

Result:

{
  "totalIssueCount": 2266
}

Object Type Example

Manifest:

requests:
  topIssue:
    query: SELECT * FROM Issues ORDER BY Id LIMIT 1
    type: Object

Result:

{
  "topIssue": {
    "id": 1,
    "ruleId": "RedundantUsingDirective",
    "level": "Warning",
    "title": "Redundancies in Code",
    "message": "Using directive is not required by the code and can be safely removed",
    "description": "Redundant using directive",
    "category": "Redundancies in Code"
  }
}

Collection Type Example

Manifest:

requests:
  issueCountByCategory:
    query: SELECT Category, COUNT(*) AS IssueCount FROM Issues GROUP BY Category
    type: Collection

Result:

{
  "issueCountByCategory": [
    {
      "category": "Common Practices and Code Improvements",
      "issueCount": 18
    },
    {
      "category": "Potential Code Quality Issues",
      "issueCount": 102
    }
  ]
}

Request Settings

Request has number of settings. Majority of them have default values but in certain cases it these parameters might be useful:

Here are few parameters with all parameters specified:

queries:
  query1:
    query: SELECT * FROM Issues WHERE RuleId = $RuleId
  query2:
    query: SELECT COUNT(*) FROM Issues WHERE Category = $Category
requests:
  inlineQuery:
    query: SELECT COUNT(*) FROM Issues WHERE Id = $IssueId
    type: Scalar
    addToOutput: true
    parameters:
      IssueId:
        value: 123
        isNull: false
        isInt: true
        isValueRef: false
    ranges:
      red:
        end: 100
      amber:
        start: 100
        end: 200
      green:
        start: 300
  queryReference:
    queryRef: query2
    type: Object
    addToOutput: true
    parameters:
      IssueId: 123

Request properties with description can be found in the following table:

Parameter Default Value Description
query Text of inline query. This property is mutualy exclusive with property queryRef. If body properties are defined query is used.
queryRef Name of query in queries collection. This property mutually exclusive with property query
type Scalar Defines result type of request.
addToOutput True Specifies if request results must be added to output. False might be used for requests which produce parameter value for other requests. By default all results are added to output
parameters Collection of parameters which needs to be passed to request.

parameters collection is optional section of request. It might be used to provide values for parametrized SQL queries. The following properties are available for parameter:

Property Default Value Description
value Value to use for parameterized query. By default all values have .NET type string
isNull False Specifies if DBNull.Value must be passed to command. This property takes priority over value property.
isInt False Performs conversion of string value into .NET type long before passing this value to SQL command
isValueRef Reference to scalar request result. Result of request (e.g. single value) is passed to parameter value.

Example

Let's assume that SQLite database contains issue statics for .NET project. Evaluation manifest is used to calculate metrics based on data stored in database. Manifest may look like this:

queries:
  issueByCategory:
    query: SELECT * FROM Issues WHERE Category = $Category LIMIT (5)
requests:
  totalIssueCount:
    query: SELECT COUNT(*) FROM Issues
    type: Scalar
  redundanciesInCode:
    queryRef: issueByCategory
    type: Collection
    parameters:
      Category: Redundancies in Code
  topIssue:
    query: SELECT * FROM Issues LIMIT (1)
    type: Object
  issueCountByCategory:
    query: SELECT Category, COUNT(*) AS IssueCount FROM Issues GROUP BY Category
    type: Collection
  projectSize:
    query: SELECT SUM(code) FROM FileDetails WHERE Language = $Category
    parameters:
      Category: C#
    ranges:
      XS:
        end: 1000
      S:
        start: 1000
        end: 10000
      M:
        start: 10000
        end: 100000
      L:
        start: 100000
        end: 500000
      XL:
        start: 500000

Evaluation result may look as follows:

{
  "totalIssueCount": 2266,
  "redundanciesInCode": [
    {
      "id": 3,
      "ruleId": "RedundantUsingDirective",
      "level": "Warning",
      "title": "Redundancies in Code",
      "message": "Using directive is not required by the code and can be safely removed",
      "description": "Redundant using directive",
      "category": "Redundancies in Code"
    },
    {
      "id": 5,
      "ruleId": "RedundantUsingDirective",
      "level": "Warning",
      "title": "Redundancies in Code",
      "message": "Using directive is not required by the code and can be safely removed",
      "description": "Redundant using directive",
      "category": "Redundancies in Code"
    },
    {
      "id": 7,
      "ruleId": "RedundantUsingDirective",
      "level": "Warning",
      "title": "Redundancies in Code",
      "message": "Using directive is not required by the code and can be safely removed",
      "description": "Redundant using directive",
      "category": "Redundancies in Code"
    },
    {
      "id": 9,
      "ruleId": "RedundantUsingDirective",
      "level": "Warning",
      "title": "Redundancies in Code",
      "message": "Using directive is not required by the code and can be safely removed",
      "description": "Redundant using directive",
      "category": "Redundancies in Code"
    }
  ],
  "topIssue": {
    "id": 1,
    "ruleId": "RedundantUsingDirective",
    "level": "Warning",
    "title": "Redundancies in Code",
    "message": "Using directive is not required by the code and can be safely removed",
    "description": "Redundant using directive",
    "category": "Redundancies in Code"
  },
  "issueCountByCategory": [
    {
      "category": "Common Practices and Code Improvements",
      "issueCount": 18
    },
    {
      "category": "Potential Code Quality Issues",
      "issueCount": 102
    },
    {
      "category": "Redundancies in Code",
      "issueCount": 240
    },
    {
      "category": "Redundancies in Symbol Declarations",
      "issueCount": 48
    }
  ],
  "projectSize": {
    "value": 1292,
    "range": "S"
  }
}

It could be seen that manifest is list of queries which are executed in the same order as they are defined in manifest. Name of each query becames a key of result in output document.

Clone this wiki locally