Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

typespec renaming #5691

Merged
5 commits merged into from
Mar 16, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
# Integrating with cadl sync and generate scripts
# Integrating with typespec sync and generate scripts

There are 2 common scripts provided for each language to be able to generate from within the language
repo and use the remote cadl definition in the spec repo.
repo and use the remote typespec definition in the spec repo.

## One time language repo setup

There are 3 things that these two scripts expect are set up in your language repo before they will run correctly.
1. Make sure your .gitignore is ignoring the TempCadlFiles
1. Make sure your .gitignore is ignoring the TempTypeSpecFiles
2. Create a common emitter-package.json for your language
3. Write the language specific hooks in Language-Settings.ps1

### TempCadlFiles
### TempTypeSpecFiles

You should add a new entry in your .gitignore for your repo so that none of these files are accidentally checked in if [cleanup](#cleanup-anchor) is turned off.

```
# .gitignore file
TempCadlFiles/
TempTypeSpecFiles/
```

### emitter-package.json
Expand All @@ -30,7 +30,7 @@ Example
{
"main": "dist/src/index.js",
"dependencies": {
"@azure-tools/cadl-csharp": "0.1.11-beta.20230123.1"
"@azure-tools/typespec-csharp": "0.1.11-beta.20230123.1"
}
}
```
Expand All @@ -50,7 +50,7 @@ Example

```powershell
function Get-dotnet-EmitterName() {
return "@azure-tools/cadl-csharp"
return "@azure-tools/typespec-csharp"
}
```

Expand All @@ -68,31 +68,31 @@ function Get-dotnet-EmitterPackageJsonPath() {

#### Get-${Language}-EmitterAdditionalOptions (Optional)

This function allows you to append additional `--option` arguments that will be passed into cadl compile. One example of this is the `emitter-output-dir`. For dotnet we want the location of the generated files to be `{projectDir}/src` however in other languages they will have other conventions. This method will take in a fully qualified path to the project directory so you can construct your relative path to that as the output.
This function allows you to append additional `--option` arguments that will be passed into typespec compile. One example of this is the `emitter-output-dir`. For dotnet we want the location of the generated files to be `{projectDir}/src` however in other languages they will have other conventions. This method will take in a fully qualified path to the project directory so you can construct your relative path to that as the output.

Example

```powershell
function Get-dotnet-EmitterAdditionalOptions([string]$projectDirectory) {
return "--option @azure-tools/cadl-csharp.emitter-output-dir=$projectDirectory/src"
return "--option @azure-tools/typespec-csharp.emitter-output-dir=$projectDirectory/src"
}
```

## Per project setup

Each project will need to have a configuration file that will tell the scripts where to find the cadl spec.
Each project will need to have a configuration file that will tell the scripts where to find the typespec spec.

### cadl-location.yaml
### tsp-location.yaml

This file should live under the project directory for each service and has the following properties

| Property | Description | IsRequired |
| --- | --- | --- |
| <a id="directory-anchor"></a> directory | The top level directory where the main.cadl for the service lives. This should be relative to the spec repo root such as `specification/cognitiveservices/OpenAI.Inference` | true |
| <a id="additionalDirectories-anchor"></a> additionalDirectories | Sometimes a cadl file will use a relative import that might not be under the main directory. In this case a single `directory` will not be enough to pull down all necessary files. To support this you can specify additional directories as a list to sync so that all needed files are synced. | false: default = null |
| <a id="commit-anchor"></a> commit | The commit sha for the version of the cadl files you want to generate off of. This allows us to have idempotence on generation until we opt into pointing at a later version. | true |
| <a id="directory-anchor"></a> directory | The top level directory where the main.tso for the service lives. This should be relative to the spec repo root such as `specification/cognitiveservices/OpenAI.Inference` | true |
| <a id="additionalDirectories-anchor"></a> additionalDirectories | Sometimes a typespec file will use a relative import that might not be under the main directory. In this case a single `directory` will not be enough to pull down all necessary files. To support this you can specify additional directories as a list to sync so that all needed files are synced. | false: default = null |
| <a id="commit-anchor"></a> commit | The commit sha for the version of the typespec files you want to generate off of. This allows us to have idempotence on generation until we opt into pointing at a later version. | true |
| <a id="repo-anchor"></a> repo | The repo this spec lives in. This should be either `Azure/azure-rest-api-specs` or `Azure/azure-rest-api-specs-pr`. Note that pr will work locally but not in CI until we add another change to handle token based auth. | true |
| <a id="cleanup-anchor"></a> cleanup | This will remove the TempCadlFiles directory after generation is complete if true otherwise this directory will be left to support local changes to the files to see how different changes would affect the generation. | false: default = true |
| <a id="cleanup-anchor"></a> cleanup | This will remove the TempTypeSpecFiles directory after generation is complete if true otherwise this directory will be left to support local changes to the files to see how different changes would affect the generation. | false: default = true |

Example

Expand All @@ -105,42 +105,42 @@ repo: Azure/azure-rest-api-specs
cleanup: false
```

## Cadl-Project-Sync.ps1
## TypeSpec-Project-Sync.ps1

This is the first script that should be called and can be found at `./eng/common/scripts/Cadl-Project-Sync.ps1`. It takes in one parameter which is the root directory of the project which is typically one layer lower than the service directory. As an example for dotnet this is `./sdk/openai/Azure.AI.OpenAI` where `openai` is the service directory and `Azure.AI.OpenAI` is the project directory.
This is the first script that should be called and can be found at `./eng/common/scripts/TypeSpec-Project-Sync.ps1`. It takes in one parameter which is the root directory of the project which is typically one layer lower than the service directory. As an example for dotnet this is `./sdk/openai/Azure.AI.OpenAI` where `openai` is the service directory and `Azure.AI.OpenAI` is the project directory.

```powershell
./eng/common/scripts/Cadl-Project-Sync.ps1 ./sdk/openai/Azure.AI.OpenAI
./eng/common/scripts/TypeSpec-Project-Sync.ps1 ./sdk/openai/Azure.AI.OpenAI
```

This script will create a `sparse-spec` folder as a sibling to the root of your current git clone. Each project that is generated will get a sub directory inside of this folder named after the project you are generating. It will then automatically filter to only the files in the [directory](#directory-anchor) defined in cadl-location.yaml, and sync to the [commit sha](#commit-anchor) defined in cadl-location.yaml.
This script will create a `sparse-spec` folder as a sibling to the root of your current git clone. Each project that is generated will get a sub directory inside of this folder named after the project you are generating. It will then automatically filter to only the files in the [directory](#directory-anchor) defined in tsp-location.yaml, and sync to the [commit sha](#commit-anchor) defined in tsp-location.yaml.

As an example if you have your language repo at `D:\git\azure-sdk-for-net` there will be a new directory `D:\git\sparse-spec\Azure.AI.OpenAI` where the sparse spec will live.

This is then copied over to your project directory so that you can make temporary changes if needed. The location will be `./{projectDir}/TempCadlFiles`. This temporary directory will be [cleaned up](#cleanup-anchor) at the end of the generate script if set in the cadl-location.yaml.
This is then copied over to your project directory so that you can make temporary changes if needed. The location will be `./{projectDir}/TempTypeSpecFiles`. This temporary directory will be [cleaned up](#cleanup-anchor) at the end of the generate script if set in the tsp-location.yaml.

## Cadl-Project-Generate.ps1
## TypeSpec-Project-Generate.ps1

This is the second script that should be called and can be found at `./eng/common/scripts/Cadl-Project-Generate.ps1`. It takes the exact same parameter as the sync script.
This is the second script that should be called and can be found at `./eng/common/scripts/TypeSpec-Project-Generate.ps1`. It takes the exact same parameter as the sync script.

```powershell
./eng/common/scripts/Cadl-Project-Generate.ps1 ./sdk/openai/Azure.AI.OpenAI
./eng/common/scripts/TypeSpec-Project-Generate.ps1 ./sdk/openai/Azure.AI.OpenAI
```

The first thing this does is clean up the npm install that might exist in `./{projectDir}/TempCadlFiles`, followed by replacing the package.json with the language static one.
The first thing this does is clean up the npm install that might exist in `./{projectDir}/TempTypeSpecFiles`, followed by replacing the package.json with the language static one.

Once this is done it will run `npm install` followed by `cadl compile` which is the standard way to generate a cadl project.
Once this is done it will run `npm install` followed by `tsp compile` which is the standard way to generate a typespec project.

The exact command that gets run is output stdout to enable debugging if needed.

We currently don't do anything to the cadl-project.yaml that gets pulled in from the spec repo to limit to just your language emitter instead we use the filter option on the command line `--emit $emitterName`. This allows you to isolate the generation to only things owned by your language so you can safely add generation dependencies in CI without needing to worry about noisy neighbors.
We currently don't do anything to the tspconfig.yaml that gets pulled in from the spec repo to limit to just your language emitter instead we use the filter option on the command line `--emit $emitterName`. This allows you to isolate the generation to only things owned by your language so you can safely add generation dependencies in CI without needing to worry about noisy neighbors.

## Build tool integration

One use case that some languages have is to have their CI regenerate the project and then do a `git diff` to validate that there are no differences. This helps detect if people modify the generated files manually. To support this its valuable to have the exact same command to generate a project regardless of whether the individual library is autorest or cadl.
One use case that some languages have is to have their CI regenerate the project and then do a `git diff` to validate that there are no differences. This helps detect if people modify the generated files manually. To support this its valuable to have the exact same command to generate a project regardless of whether the individual library is autorest or typespec.

To achieve this each language will have their own idiomatic tool set but whatever that toolset is can check to see if a cadl-location.yaml file exists, and if it does they can call the Cadl-Project-Sync.ps1 and Cadl-Project-Generate.ps1 scripts, otherwise they can call the autorest command they call today for all other libraries.
To achieve this each language will have their own idiomatic tool set but whatever that toolset is can check to see if a tsp-location.yaml file exists, and if it does they can call the TypeSpec-Project-Sync.ps1 and TypeSpec-Project-Generate.ps1 scripts, otherwise they can call the autorest command they call today for all other libraries.

In dotnet this is achieved by running `dotnet build /t:GenerateCode` regardless of which type of project it is the correct commands get called and it remains consistent and idiomatic to the language. In other languages this could be `npm generate` or `python generate.py` to do the same.

Since the generate script simply is a wrapper for `npm install` and `cadl compile` you can still run those commands directly manually after the sync if you want to instead.
Since the generate script simply is a wrapper for `npm install` and `tsp compile` you can still run those commands directly manually after the sync if you want to instead.
101 changes: 101 additions & 0 deletions eng/common/scripts/TypeSpec-Project-Generate.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# For details see https://github.com/Azure/azure-sdk-tools/blob/main/doc/common/TypeSpec-Project-Scripts.md

[CmdletBinding()]
param (
[Parameter(Position=0)]
[ValidateNotNullOrEmpty()]
[string] $ProjectDirectory,
[Parameter(Position=1)]
[string] $typespecAdditionalOptions ## addtional typespec emitter options, separated by semicolon if more than one, e.g. option1=value1;option2=value2
)

$ErrorActionPreference = "Stop"
. $PSScriptRoot/Helpers/PSModule-Helpers.ps1
. $PSScriptRoot/common.ps1
Install-ModuleIfNotInstalled "powershell-yaml" "0.4.1" | Import-Module

function NpmInstallForProject([string]$workingDirectory) {
Push-Location $workingDirectory
try {
$currentDur = Resolve-Path "."
Write-Host "Generating from $currentDur"

if (Test-Path "package.json") {
Remove-Item -Path "package.json" -Force
}

if (Test-Path ".npmrc") {
Remove-Item -Path ".npmrc" -Force
}

if (Test-Path "node_modules") {
Remove-Item -Path "node_modules" -Force -Recurse
}

if (Test-Path "package-lock.json") {
Remove-Item -Path "package-lock.json" -Force
}

#default to root/eng/emitter-package.json but you can override by writing
#Get-${Language}-EmitterPackageJsonPath in your Language-Settings.ps1
$replacementPackageJson = "$PSScriptRoot/../../emitter-package.json"
if (Test-Path "Function:$GetEmitterPackageJsonPathFn") {
$replacementPackageJson = &$GetEmitterPackageJsonPathFn
}

Write-Host("Copying package.json from $replacementPackageJson")
Copy-Item -Path $replacementPackageJson -Destination "package.json" -Force
npm install --no-lock-file
if ($LASTEXITCODE) { exit $LASTEXITCODE }
}
finally {
Pop-Location
}
}

$resolvedProjectDirectory = Resolve-Path $ProjectDirectory
$emitterName = &$GetEmitterNameFn
$typespecConfigurationFile = Resolve-Path "$ProjectDirectory/tsp-location.yaml"

Write-Host "Reading configuration from $typespecConfigurationFile"
$configuration = Get-Content -Path $typespecConfigurationFile -Raw | ConvertFrom-Yaml

$specSubDirectory = $configuration["directory"]
$innerFolder = Split-Path $specSubDirectory -Leaf

$tempFolder = "$ProjectDirectory/TempTypeSpecFiles"
$npmWorkingDir = Resolve-Path $tempFolder/$innerFolder
$mainTypeSpecFile = If (Test-Path "$npmWorkingDir/client.*") { Resolve-Path "$npmWorkingDir/client.*" } Else { Resolve-Path "$npmWorkingDir/main.*"}

try {
Push-Location $npmWorkingDir
NpmInstallForProject $npmWorkingDir

if ($LASTEXITCODE) { exit $LASTEXITCODE }

if (Test-Path "Function:$GetEmitterAdditionalOptionsFn") {
$emitterAdditionalOptions = &$GetEmitterAdditionalOptionsFn $resolvedProjectDirectory
if ($emitterAdditionalOptions.Length -gt 0) {
$emitterAdditionalOptions = " $emitterAdditionalOptions"
}
}
$typespecCompileCommand = "npx tsp compile $mainTypeSpecFile --emit $emitterName$emitterAdditionalOptions"
if ($typespecAdditionalOptions) {
$options = $typespecAdditionalOptions.Split(";");
foreach ($option in $options) {
$typespecCompileCommand += " --option $emitterName.$option"
}
}
Write-Host($typespecCompileCommand)
Invoke-Expression $typespecCompileCommand

if ($LASTEXITCODE) { exit $LASTEXITCODE }
}
finally {
Pop-Location
}

$shouldCleanUp = $configuration["cleanup"] ?? $true
if ($shouldCleanUp) {
Remove-Item $tempFolder -Recurse -Force
}
chunyu3 marked this conversation as resolved.
Show resolved Hide resolved
Loading