[dev-tool] Inline snippet extraction. #24536

witemple-msft · 2023-01-20T22:01:34Z

This PR implements a new dev-tool command: dev-tool run update-snippets.

This command looks for code fences in markdown files and JSDoc comments, and updates them with the contents of test methods in a file named snippets.spec.ts.

For example, the following fence indicates that the contents of a test named "new_configurationclient" should be used:

```js snippet:new_configurationclient
```

After running dev-tool run update-snippets, the contents of the snippet will be populated:

```js snippet:new_configurationclient
import { ConfigurationClient } from "@azure/template";
import { DefaultAzureCredential } from "@azure/identity";

const client = new ConfigurationClient(
  "<app configuration endpoint>",
  new DefaultAzureCredential()
);
```

To accomplish this, the command uses the TypeScript compiler API to extract and transpile snippets from snippets.spec.ts. Snippets are the contents of calls to the it function. If syntax with the shape it(<literal string>, <function with block>) appears in snippets.spec.ts, it will be considered a snippet that is valid for injection.

("Function with block" means either a function () { ... } expression or an arrow function with a block on the arrow side (() => { ... }). An arrow function that has an expression on the right hand side (() => (...)) will not be recognized.)

For example:

  it("new_configurationclient", function () {
    // @ts-ignore
    const client = new ConfigurationClient(
      process.env.ENDPOINT ?? "<app configuration endpoint>",
      new DefaultAzureCredential()
    );
  });

The transpiler automatically "cleans" and validates the snippet using similar techniques as the sample transpiler. As a result, it enforces the same syntactic rules that the sample transpiler does. In addition to those, it removes references to process.env (if an alternative is specified), removes compiler pragmas like // @ts-ignore, and automatically inserts imports for symbols that the snippet uses. So in the above snippet, imports for ConfigurationClient and DefaultAzureCredential are required, automatically detected, and injected into the resulting snippet.

Snippets without snippet:${name} tags are errors when using this command, so a package must be fully migrated to use it.

mpodwysocki · 2023-01-20T23:03:40Z

common/tools/dev-tool/src/util/prettier.ts

+
+const prettierOptions: prettier.Options = {
+  // eslint-disable-next-line @typescript-eslint/no-var-requires
+  ...(require("../../../eslint-plugin-azure-sdk/prettier.json") as prettier.Options),


Do we want to use require here or should we just be using fs to read the file?

I guess this is just how I did it previously in the first iterations of the dev-tool sample processor. It's just simpler but I'm happy to do a readFileSync into JSON.parse instead. Without top-level async in all our node targets I can't easily use async file IO in the module body, so that's why I did it this way.

mpodwysocki · 2023-01-20T23:05:22Z

sdk/template/template/README.md

@@ -75,16 +75,27 @@ Create a section for each top-level service concept you want to explain.

 Create several code examples for how someone would use your library to accomplish a common task with the service.

+```js snippet:FooBar


Do we want to keep these here or have real samples that are related to the template project?

This will be removed and replaced with something more realistic.

mpodwysocki · 2023-01-20T23:06:06Z

sdk/template/template/test/snippets.spec.ts

@@ -0,0 +1,29 @@
+import { DefaultAzureCredential } from "@azure/identity";
+import { setLogLevel } from "@azure/logger";
+import { ConfigurationClient } from "../src";


Should we change this already to @azure/template since it should be mapped via the tsconfig file?

That's a good question. While the tsconfig handles the mapping at compile time, runtime support for mapping these imports is provided by dev-tool, but currently dev-tool can only host samples. I have a PR to add support for hosting the tests to dev-tool, but it breaks some tests for reasons that I've found difficult to understand.

Being able to import from "@azure/template" in tests like you're suggesting is blocked on that PR.

mpodwysocki · 2023-01-20T23:06:37Z

sdk/template/template/test/snippets.spec.ts

+    );
+  });
+
+  it("FooBar", () => {


Should we keep these as markers for testing purposes or should we have samples strictly for template?

### Packages impacted by this PR - [dev-tool] ### Issues associated with this PR ### Describe the problem that is addressed by this PR Adds a design document on how we are going to do snippet extraction for READMEs and source code. ### What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen? ### Are there test cases added in this PR? _(If not, why?)_ ### Provide a list of related PRs _(if any)_ - #24536 ### Command used to generate this PR:**_(Applicable only to SDK release request PRs)_ ### Checklists - [ ] Added impacted package name to the issue description - [ ] Does this PR needs any fixes in the SDK Generator?** _(If so, create an Issue in the [Autorest/typescript](https://github.com/Azure/autorest.typescript) repository and link it here)_ - [ ] Added a changelog (if necessary) --------- Co-authored-by: Will Temple <witemple@microsoft.com>

ghost · 2023-03-31T10:04:13Z

Hi @witemple-msft. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

ghost · 2023-04-07T17:03:11Z

Hi @witemple-msft. Thank you for your contribution. Since there hasn't been recent engagement, we're going to close this out. Feel free to respond with a comment containing "/reopen" if you'd like to continue working on these changes. Please be sure to use the command to reopen or remove the "no-recent-activity" label; otherwise, this is likely to be closed again with the next cleanup pass.

github-actions · 2023-06-09T05:03:20Z

Hi @witemple-msft. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

github-actions · 2023-06-16T08:34:29Z

Hi @witemple-msft. Thank you for your contribution. Since there hasn't been recent engagement, we're going to close this out. Feel free to respond with a comment containing /reopen if you'd like to continue working on these changes. Please be sure to use the command to reopen or remove the no-recent-activity label; otherwise, this is likely to be closed again with the next cleanup pass.

@ts-ignore

### Packages impacted by this PR - @azure/dev-tool ### Issues associated with this PR ### Describe the problem that is addressed by this PR This PR implements a new dev-tool command: `dev-tool run update-snippets`. This command looks for code fences in markdown files and JSDoc comments, and updates them with the contents of test methods in a file named `snippets.spec.ts`. For example, the following fence indicates that the contents of a test named "new_configurationclient" should be used: ```` ```js snippet:new_configurationclient ``` ```` After running `dev-tool run update-snippets`, the contents of the snippet will be populated: ```` ```js snippet:new_configurationclient import { ConfigurationClient } from "@azure/template"; import { DefaultAzureCredential } from "@azure/identity"; const client = new ConfigurationClient( "<app configuration endpoint>", new DefaultAzureCredential() ); ``` ```` To accomplish this, the command uses the TypeScript compiler API to extract and transpile snippets from `snippets.spec.ts`. Snippets are the contents of calls to the `it` function. If syntax with the shape `it(<literal string>, <function with block>)` appears in `snippets.spec.ts`, it will be considered a snippet that is valid for injection. ("Function with block" means either a `function () { ... }` expression or an arrow function with a block on the arrow side (`() => { ... }`). An arrow function that has an expression on the right hand side (`() => (...)`) will not be recognized.) For example: ```ts it("new_configurationclient", function () { // @ts-ignore const client = new ConfigurationClient( process.env.ENDPOINT ?? "<app configuration endpoint>", new DefaultAzureCredential() ); }); ``` The transpiler automatically "cleans" and validates the snippet using similar techniques as the sample transpiler. As a result, it enforces the same syntactic rules that the sample transpiler does. In addition to those, it removes references to `process.env` (if an alternative is specified), removes compiler pragmas like `// @ts-ignore`, and automatically inserts imports for symbols that the snippet uses. So in the above snippet, imports for `ConfigurationClient` and `DefaultAzureCredential` are required, automatically detected, and injected into the resulting snippet. Snippets without `snippet:${name}` tags are _errors_ when using this command, so a package must be fully migrated to use it. ### What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen? ### Are there test cases added in this PR? _(If not, why?)_ ### Provide a list of related PRs _(if any)_ - #24536 ### Command used to generate this PR:**_(Applicable only to SDK release request PRs)_ ### Checklists - [ ] Added impacted package name to the issue description - [ ] Does this PR needs any fixes in the SDK Generator?** _(If so, create an Issue in the [Autorest/typescript](https://github.com/Azure/autorest.typescript) repository and link it here)_ - [ ] Added a changelog (if necessary)

[dev-tool] Snippet extraction.

516fe2b

witemple-msft added EngSys This issue is impacting the engineering system. dev-tool Issues related to the Azure SDK for JS dev-tool labels Jan 20, 2023

witemple-msft requested a review from mpodwysocki January 20, 2023 22:01

witemple-msft self-assigned this Jan 20, 2023

witemple-msft requested review from mikeharder, ckairen, deyaaeldeen, weshaggard and benbp as code owners January 20, 2023 22:01

mpodwysocki mentioned this pull request Jan 20, 2023

[dev tool] Add design docs for README snippets #24537

Merged

3 tasks

mpodwysocki reviewed Jan 20, 2023

View reviewed changes

Added some more thorough documentation.

f0d2d7a

ghost added the no-recent-activity There has been no recent activity on this issue. label Mar 31, 2023

ghost closed this Apr 7, 2023

witemple-msft reopened this Apr 7, 2023

witemple-msft removed the no-recent-activity There has been no recent activity on this issue. label Apr 7, 2023

github-actions bot added the no-recent-activity There has been no recent activity on this issue. label Jun 9, 2023

github-actions bot closed this Jun 16, 2023

mpodwysocki mentioned this pull request Sep 17, 2024

[dev-tool] Add snippets extraction #31143

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dev-tool] Inline snippet extraction. #24536

[dev-tool] Inline snippet extraction. #24536

witemple-msft commented Jan 20, 2023

mpodwysocki Jan 20, 2023

witemple-msft Jan 23, 2023

mpodwysocki Jan 20, 2023

witemple-msft Jan 23, 2023

mpodwysocki Jan 20, 2023

witemple-msft Jan 23, 2023

mpodwysocki Jan 20, 2023

ghost commented Mar 31, 2023

ghost commented Apr 7, 2023

github-actions bot commented Jun 9, 2023

github-actions bot commented Jun 16, 2023

		@@ -75,16 +75,27 @@ Create a section for each top-level service concept you want to explain.

		Create several code examples for how someone would use your library to accomplish a common task with the service.

		```js snippet:FooBar

[dev-tool] Inline snippet extraction. #24536

[dev-tool] Inline snippet extraction. #24536

Conversation

witemple-msft commented Jan 20, 2023

mpodwysocki Jan 20, 2023

Choose a reason for hiding this comment

witemple-msft Jan 23, 2023

Choose a reason for hiding this comment

mpodwysocki Jan 20, 2023

Choose a reason for hiding this comment

witemple-msft Jan 23, 2023

Choose a reason for hiding this comment

mpodwysocki Jan 20, 2023

Choose a reason for hiding this comment

witemple-msft Jan 23, 2023

Choose a reason for hiding this comment

mpodwysocki Jan 20, 2023

Choose a reason for hiding this comment

ghost commented Mar 31, 2023

ghost commented Apr 7, 2023

github-actions bot commented Jun 9, 2023

github-actions bot commented Jun 16, 2023