Skip to content

Latest commit

 

History

History
115 lines (64 loc) · 8.38 KB

Fitness-for-purpose-overview.md

File metadata and controls

115 lines (64 loc) · 8.38 KB
Raw

Table of contents

Appendix: Fitness for purpose: An overview

Introduction

This Appendix defines a "playbook" for ensuring that a SARIF log file contains the information it needs so it can be used for any particular purpose. We refer to a SARIF log file that contains the necessary information as being fit for purpose.

SARIF log files can be used in many ways. For example:

  • Viewing: An end user can examine a log file in a SARIF "viewer", such as this web-based viewer.

  • Result matching: A build system can run static analysis tools as part of a nightly build. It can compare the SARIF logs from the current build with corresponding logs from the previous build, and notify the team of any new results. We refer to the process of identifying results that have appeared, disappeared, or changed from one run to another as result matching.

  • Automatic bug filing: A build system can use result matching to identify new results, and then automatically file bugs for them.1

SARIF defines a wide variety of objects and properties, most of which are optional. Depending on what the log file will be used for, various subsets of those properties are important.

The playbook

The purpose of the playbook is:

  • To enable SARIF producers to create log files that are fit for a specific purpose.
  • To enable SARIF consumers to confirm that the log files they receive are fit for purpose before attempting to use them for that purpose.

For example, a build system might need to produce log files that are suitable for automatic bug filing, and an automatic bug filing system would need to ensure that the log files it receives are suitable for automatic bug filing before attempting to file bugs from them.

The playbook consists of the following steps.

Step 1: Define and document a standard for the usage scenario

Specify the SARIF objects and properties necessary to support the scenario.

If necessary, specify any constraints on the required properties beyond those defined by SARIF itself. For example, it might be necessary for a URI-valued property to contain a relative reference rather than an absolute URI.

Explain the value of each object, property, and constraint in the scenario.

Step 2: Identify and/or implement analysis rules to verify fitness for purpose

The SARIF Multitool offers a validate command that runs a set of analysis rules on a SARIF log file, and produces another log file with the results:

sarif validate MyFile.sarif --output MyFile-validation.sarif

The complete set of existing analysis rules is available in the src/Sarif.Multitool/rules directory in the microsoft/sarif-sdk repo.

For each element of the fitness standard identified in Step 1, identify a validation rule that enforces it. If no such rule exists, then:

  • If the rule is generally useful, contribute a new rule to the sarif-sdk repo.
  • If the rule is specific to a single, proprietary system, implement the rule in a plug-in rules assembly.2

If you are contributing a new rule, consider whether the rule is useful in all scenarios (in which case it should be enabled by default) or is useful only in this particular scenario (in which case it should be disabled by default).

In some cases, it might not be practical to implement an analysis rule for an element of the standard. For example, the standard might impose certain quality requirements on user-facing messages that are difficult to validate automatically.

Document the set of analysis rules that validates fitness for purpose. Call out any fitness criteria for which a rule cannot be written.

Step 3: Provide fixes for issues identified by the analysis rules

Provide a programmatic mechanism for fixing (as far as possible) the issues identified by the analysis rules listed in Step 2.

The SARIF Multitool offers a rewrite command that can perform a variety of operations on a SARIF log file, such as populating optional SARIF properties (we refer to this operation as enrichment). For example, you can embed the contents of the analyzed files into the SARIF log file:

sarif rewrite MyFile.sarif --insert TextFiles --output MyFile-enriched.sarif

Identify any existing rewrite options that help to make a SARIF log file fit for purpose. These might include one or more of the arguments to the --insert option, or other existing options to the rewrite command.

If there is no command line option that performs the necessary operations, consider contributing a configurable API to the SARIF SDK and integrating it into the rewrite command -- either as an additional argument to the --insert option, or as a separate command line option.3

Document the complete set of rewrite options that fix the issues identified by the analysis rules. If any other programmatic or manual steps are necessary to make the log file fit for purpose, document those as well.

Step 4: Provide a configuration file that enables just the relevant rules.

Provide an XML configuration file that

  • Enables the analysis rules identified in Step 1.
  • Configures the rules, if necessary, with appropriate parameter values.
  • Disables all other rules with rule ids SARIF2000 and above.4

If appropriate, contribute the configuration file to the policies directory of the microsoft/sarif-sdk repo.

You can use policies/gitub-dsp.config.xml as a model.5

To validate a SARIF file according to a rule configuration, run this command (for example):

sarif validate MyFile.sarif --config policies\github-dsp.config.xml --verbose --output MyFile-validation.sarif

Note the --verbose option. By default, the validate command will not produce results for rules whose level is "note", even if the rule is enabled.6 When --verbose is specified, the validator's output includes "note"-level results.

Summary

That's it. SARIF producers can use the sarif rewrite command defined in Step 3 to produce SARIF log files that are fit for purpose. SARIF consumers can use the sarif validate command with the configuration file defined in Step 4 to verify that the files it receives are fit for purpose before it processes them further. During development, SARIF producers can do the same to ensure that their rewrite operation works properly.

Notes

1. The system doesn't need to file a separate bug for each result. It might file one bug per source file, or even one bug per tool run.

2. At the time of writing, plug-in rule assemblies are not supported. Issue microsoft/sarif-sdk#1868, "Allow validation command to accept plug-ins with additional validation rules", tracks this work.

3. At the time of writing, we consider the rewrite command the appropriate place to integrate operations that make SARIF log files fit for purpose. It is possible that a future version of the SARIF Multitool will provide a different way of accessing this functionality.

4. Analysis rules in the range SARIF1001-1999 represent absolute requirements of the SARIF standard. They are enabled by default and should never be disabled.

5. At the time of writing, this config file does not follow the guidance of disabling non-relevant 2000-level rules. Issue microsoft/sarif-sdk#2039, "Disable non-relevant 2000-level rules in DSP config files", tracks this work.

6. We're considering changing this behavior, which in the past has caused me to overlook results I actually cared about.

Table of contents