Changed files includes excluded paths

[BernieWhite]

Description of the issue

Changes files (with Input.IgnoreUnchangedPath) does not seem to be honouring Input.PathIgnore.

[ReneRebsdorf]

@BernieWhite wanted to search for related issues before posting this, then I saw this issue is marked for v3.

So, just curious if this is modified in the anticipated v3?:

I am trying to implement input.ignoreUnchangedPath and repository.baseRef
Ideally we would want PRs (i.e. non-baseRef?) to only scan for changed files.
When the linter is ran in main (e.g. through a pipeline schedule) the entire repo should be scanned.

The goal is to get fast PRs, that are not blocked due to "irrelevant" changes, but still keep up to date on changes to our baselines.

I have tried setting ps-rule.yaml as per my description above, but also tried setting ignoreUnchangedPath as an env var, but that still caused all files to be scanned. I did not set the baseRef as I suspect you pick it up based on variables as per the docs mentioning it (docs doesnt mention which variable it look for though which would be great).

So all in all: Am I trying to do something that is not supported? am I getting something wrong? Are there bugs relating to inheritence of values based on env vars and .yaml files? (We are seeing problems merging hashtables and yaml files, but that is a separate issue(.

There seems to be a lot of modifications coming with v3, so I am hesitant to spend time fixing on v2 (ETA on v3?) :-)

[BernieWhite]

Thanks @ReneRebsdorf. It's not planned for v3.0.0, which we're trying to finalize. The issue don't have any priority currently, however we can look to slot it in for v3.1.0 or v3.2.0 if it's customer impacting.

To help clarify this issue, only changed files are processed by PSRule as expected but the bug is: files that have changes but would normally be excluded by Input.PathIgnore are still processed.

---

For your issue a key thing to check would be to make sure the full git history is fetched (noticing that this is not in the docs and the example in the docs is omits this). It's required to compare branches, because shallow fetching is the default.

https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps-checkout?view=azure-pipelines#shallow-fetch

For ADO you need to use the fetchDepth: 0 on your checkout step, and add an explicit checkout step if you haven't added it. for example:

  # Checkout the repository
  - checkout: self
    fetchDepth: 0

Let me know if that doesn't solve the issue, and we can raise a separate issue, but I'll add an item to fix that omission in the docs. Thanks.

[ReneRebsdorf]

@BernieWhite
As always, thank you for the fast and detailed response

We use fetch depth in other stages, so not sure why I didn't think of using 0 here, which is obviously required 😊

Just to see clarification: would I get full scanning on main and only change file scanning on Prs, if I set both options in the yaml file directly, or should I use environment variables? And are yaml+env vars supported simultaneously?
If not, how do you recommend achieving this, or do you have a different approach we should consider?

[BernieWhite]

@ReneRebsdorf PSRule supports environment variables and YAML together (for almost all options). The environment variable overrides the YAML setting if the settings conflict.

So, you should use the Input.IgnoreUnchangedPath = true option for PR builds.

For full builds you should not set this option or set the option to false so all files are scanned.

An easy option for most cases like you have suggested is to omit the option from the ps-rule.yaml and add the PSRULE_INPUT_IGNOREUNCHANGEDPATH = true environment variable to the PR build.

But if you wanted developers to be able to run checks against their branch locally with only changes files then it might be better to do the reverse, since PSRule in VSCode will use ps-rule.yaml and setting local environment variables on a dev machine is additional setup.

[ReneRebsdorf]

@ReneRebsdorf PSRule supports environment variables and YAML together (for almost all options). The environment variable overrides the YAML setting if the settings conflict.

So, you should use the Input.IgnoreUnchangedPath = true option for PR builds.

For full builds you should not set this option or set the option to false so all files are scanned.

An easy option for most cases like you have suggested is to omit the option from the ps-rule.yaml and add the PSRULE_INPUT_IGNOREUNCHANGEDPATH = true environment variable to the PR build.

But if you wanted developers to be able to run checks against their branch locally with only changes files then it might be better to do the reverse, since PSRule in VSCode will use ps-rule.yaml and setting local environment variables on a dev machine is additional setup.

Thanks!

Yeah I think I will setup local files to assume to only test pr changes, and then use a variable in the Ci pipeline to control overwriting with an env var.