-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into feature/ta-v3.2-preview.2
- Loading branch information
Showing
3,066 changed files
with
512,875 additions
and
430,254 deletions.
The diff you're trying to view is too large. We only load the first 3000 changed files.
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,177 @@ | ||
| # Guide for migrating to the test proxy from vcrpy | ||
|
|
||
| This guide describes the changes that service SDKs should make to their test frameworks in order to take advantage of | ||
| the Azure SDK test proxy. | ||
|
|
||
| Documentation of the motivations and goals of the test proxy can be found [here][general_docs] in the azure-sdk-tools | ||
| GitHub repository, and documentation of how to set up and use the proxy can be found [here][detailed_docs]. | ||
|
|
||
| ## Update existing tests | ||
|
|
||
| ### Current test structure | ||
|
|
||
| Test classes currently inherit from AzureTestCase, and test methods can optionally use decorators: | ||
|
|
||
| ```py | ||
| from devtools_testutils import AzureTestCase | ||
|
|
||
| class TestExample(AzureTestCase): | ||
|
|
||
| def test_example(self): | ||
| ... | ||
|
|
||
| @ExamplePreparer() | ||
| def test_example_with_preparer(self): | ||
| ... | ||
| ``` | ||
|
|
||
| ### New test structure | ||
|
|
||
| To use the proxy, test classes should inherit from AzureRecordedTestCase and recorded test methods should use a | ||
| RecordedByProxy decorator: | ||
|
|
||
| ```py | ||
| from devtools_testutils import AzureRecordedTestCase, RecordedByProxy | ||
|
|
||
| class TestExample(AzureRecordedTestCase): | ||
|
|
||
| @RecordedByProxy | ||
| def test_example(self): | ||
| ... | ||
|
|
||
| @ExamplePreparer() | ||
| @RecordedByProxy | ||
| def test_example_with_preparer(self): | ||
| ... | ||
| ``` | ||
|
|
||
| For async tests, import the RecordedByProxyAsync decorator from `devtools_testutils.aio` and use it in the same | ||
| way as RecordedByProxy. | ||
|
|
||
| > **Note:** since AzureRecordedTestCase doesn't inherit from `unittest.TestCase`, test class names need to start | ||
| > with "Test" in order to be properly collected by pytest by default. For more information, please refer to | ||
| > [pytest's documentation][pytest_collection]. | ||
| ## Run the tests | ||
|
|
||
| ### Perform one-time setup | ||
|
|
||
| The test proxy is made available for your tests via a Docker container. Some tests require an SSL connection to work, so | ||
| the Docker image used for the container has a certificate imported that you need to trust on your machine. Instructions | ||
| on how to do so can be found [here][proxy_cert_docs]. | ||
|
|
||
| ### Start the proxy server | ||
|
|
||
| There is a [PowerShell script][docker_start_proxy] in `eng/common/testproxy` that will fetch the proxy Docker image if | ||
| you don't already have it, and will start or stop a container running the image for you. You can run the following | ||
| command from the root of the `azure-sdk-for-python` directory to start the container whenever you want to make the test | ||
| proxy available for running tests: | ||
|
|
||
| ```powershell | ||
| .\eng\common\testproxy\docker-start-proxy.ps1 "start" | ||
| ``` | ||
|
|
||
| Note that the proxy is available as long as the container is running. In other words, you don't need to start and | ||
| stop the container for each test run or between tests for different SDKs. You can run the above command in the morning | ||
| and just stop the container whenever you'd like. To stop the container, run the same command but with `"stop"` in place | ||
| of `"start"`. In the future, the proxy container will be set up and started automatically when tests are run, and | ||
| starting it manually will be optional. | ||
|
|
||
| For more details on proxy startup, please refer to the [proxy documentation][detailed_docs]. | ||
|
|
||
| ### Record or play back tests | ||
|
|
||
| Configuring live and playback tests is done with the `AZURE_TEST_RUN_LIVE` environment variable. When this variable is | ||
| set to "true" or "yes", live tests will run and produce recordings. When this variable is set to "false" or "no", or | ||
| not set at all, tests will run in playback mode and attempt to match existing recordings. | ||
|
|
||
| Recordings for a given package will end up in that package's `/tests/recordings` directory, just like they currently | ||
| do. | ||
|
|
||
| > **Note:** at this time, support for configuring live or playback tests with a `testsettings_local.cfg` file has been | ||
| > deprecated in favor of using just `AZURE_TEST_RUN_LIVE`. | ||
| ### Register sanitizers | ||
|
|
||
| Since the test proxy doesn't use [`vcrpy`][vcrpy], tests don't use a scrubber to sanitize values in recordings. | ||
| Instead, sanitizers (as well as matchers and transforms) can be registered on the proxy as detailed in | ||
| [this][sanitizers] section of the proxy documentation. At the time of writing, sanitizers can be registered via the | ||
| `add_sanitizer` method in `devtools_testutils`. | ||
|
|
||
| Sanitizers, matchers, and transforms remain registered until the proxy container is stopped, so for any sanitizers that | ||
| are shared by different tests, using a session fixture declared in a `conftest.py` file is recommended. Please refer to | ||
| [pytest's scoped fixture documentation][pytest_fixtures] for more details. | ||
|
|
||
| For example, to sanitize URIs in recordings, you can set up a URI sanitizer for all tests in the pytest session by | ||
| adding something like the following in the package's `conftest.py` file: | ||
|
|
||
| ```python | ||
| from devtools_testutils import add_sanitizer | ||
|
|
||
| # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method | ||
| @pytest.fixture(scope="session", autouse=True) | ||
| def sanitize_uris(): | ||
| add_sanitizer(ProxyRecordingSanitizer.URI, value="fakeendpoint") | ||
| ``` | ||
|
|
||
| `add_sanitizer` accepts a sanitizer, matcher, or transform type from the ProxyRecordingSanitizer enum as a required | ||
| parameter. Keyword-only arguments can be provided to customize the sanitizer; for example, in the snippet above, any | ||
| request URIs that match the default URI regular expression will have their domain name replaced with "fakeendpoint". A | ||
| request made to `https://tableaccount.table.core.windows.net` will be recorded as being made to | ||
| `https://fakeendpoint.table.core.windows.net`. | ||
|
|
||
| ## Implementation details | ||
|
|
||
| ### What does the test proxy do? | ||
|
|
||
| The gist of the test proxy is that it stands in between your tests and the service. What this means is that test | ||
| requests which would usually go straight to the service should instead point to the locally-hosted test proxy. | ||
|
|
||
| For example, if an operation would typically make a GET request to | ||
| `https://fakeazsdktestaccount.table.core.windows.net/Tables`, that operation should now be sent to | ||
| `https://localhost:5001/Tables` instead. The original endpoint should be stored in an `x-recording-upstream-base-uri` -- | ||
| the proxy will send the original request and record the result. | ||
|
|
||
| The RecordedByProxy and RecordedByProxyAsync decorators patch test requests to do this for you. | ||
|
|
||
| ### How does the test proxy know when and what to record or play back? | ||
|
|
||
| This is achieved by making POST requests to the proxy server that say whether to start or stop recording or playing | ||
| back, as well as what test is being run. | ||
|
|
||
| To start recording a test, the server should be primed with a POST request: | ||
|
|
||
| ``` | ||
| URL: https://localhost:5001/record/start | ||
| headers { | ||
| "x-recording-file": "<path-to-test>/recordings/<testfile>.<testname>" | ||
| } | ||
| ``` | ||
|
|
||
| This will return a recording ID in an `x-recording-id` header. This ID should be sent as an `x-recording-id` header in | ||
| all further requests during the test. | ||
|
|
||
| After the test has finished, a POST request should be sent to indicate that recording is complete: | ||
|
|
||
| ``` | ||
| URL: https://localhost:5001/record/stop | ||
| headers { | ||
| "x-recording-id": "<x-recording-id>" | ||
| } | ||
| ``` | ||
|
|
||
| Running tests in playback follows the same pattern, except that requests will be sent to `/playback/start` and | ||
| `/playback/stop` instead. A header, `x-recording-mode`, should be set to `record` for all requests when recording and | ||
| `playback` when playing recordings back. More details can be found [here][detailed_docs]. | ||
|
|
||
| The RecordedByProxy and RecordedByProxyAsync decorators send the appropriate requests at the start and end of each test | ||
| case. | ||
|
|
||
| [detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md | ||
| [docker_start_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/testproxy/docker-start-proxy.ps1 | ||
| [general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md | ||
| [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/trusting-cert-per-language.md | ||
| [pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery | ||
| [pytest_fixtures]: https://docs.pytest.org/latest/fixture.html#scope-sharing-fixtures-across-classes-modules-packages-or-session | ||
| [sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#session-and-test-level-transforms-sanitiziers-and-matchers | ||
| [vcrpy]: https://vcrpy.readthedocs.io |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
19 changes: 19 additions & 0 deletions
19
eng/common/pipelines/templates/steps/detect-api-changes.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| parameters: | ||
| ArtifactPath: $(Build.ArtifactStagingDirectory) | ||
| Artifacts: [] | ||
|
|
||
| steps: | ||
| - task: Powershell@2 | ||
| inputs: | ||
| filePath: $(Build.SourcesDirectory)/eng/common/scripts/Detect-Api-Changes.ps1 | ||
| arguments: > | ||
| -ArtifactList ('${{ convertToJson(parameters.Artifacts) }}' | ConvertFrom-Json | Select-Object Name) | ||
| -ArtifactPath ${{parameters.ArtifactPath}} | ||
| -CommitSha '$(Build.SourceVersion)' | ||
| -BuildId $(Build.BuildId) | ||
| -PullRequestNumber $(System.PullRequest.PullRequestNumber) | ||
| -RepoFullName $(Build.Repository.Name) | ||
| pwsh: true | ||
| workingDirectory: $(Pipeline.Workspace) | ||
| displayName: Detect API changes | ||
| condition: and(succeededOrFailed(), eq(variables['Build.Reason'],'PullRequest')) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| parameters: | ||
| ExclusionDataBaseFileName: '' | ||
| TargetDirectory: '' | ||
| PublishAnalysisLogs: false | ||
| PoliCheckBlobSAS: "$(azuresdk-policheck-blob-SAS)" | ||
| ExclusionFilePath: "$(Build.SourcesDirectory)/eng/guardian-tools/policheck/PolicheckExclusions.xml" | ||
|
|
||
| steps: | ||
| - pwsh: | | ||
| azcopy copy "https://azuresdkartifacts.blob.core.windows.net/policheck/${{ parameters.ExclusionDataBaseFileName }}.mdb?${{ parameters.PoliCheckBlobSAS }}" ` | ||
| "$(Build.BinariesDirectory)" | ||
| displayName: 'Download PoliCheck Exclusion Database' | ||
| - task: securedevelopmentteam.vss-secure-development-tools.build-task-policheck.PoliCheck@2 | ||
| displayName: 'Run PoliCheck' | ||
| inputs: | ||
| targetType: F | ||
| targetArgument: "$(Build.SourcesDirectory)/${{ parameters.TargetDirectory }}" | ||
| result: PoliCheck.sarif | ||
| optionsFC: 0 | ||
| optionsXS: 1 | ||
| optionsPE: 1|2|3|4 | ||
| optionsRulesDBPath: "$(Build.BinariesDirectory)/${{ parameters.ExclusionDataBaseFileName }}.mdb" | ||
| optionsUEPATH: ${{ parameters.ExclusionFilePath }} | ||
|
|
||
| - task: securedevelopmentteam.vss-secure-development-tools.build-task-postanalysis.PostAnalysis@2 | ||
| displayName: 'Post Analysis (PoliCheck)' | ||
| inputs: | ||
| GdnBreakAllTools: false | ||
| GdnBreakGdnToolPoliCheck: true | ||
| GdnBreakGdnToolPoliCheckSeverity: Warning | ||
| continueOnError: true | ||
|
|
||
| - ${{ if eq(parameters.PublishAnalysisLogs, 'true') }}: | ||
| - task: securedevelopmentteam.vss-secure-development-tools.build-task-publishsecurityanalysislogs.PublishSecurityAnalysisLogs@3 | ||
| displayName: 'Publish Security Analysis Logs' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.