Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Anuj] Adding Fabric CI/CD sample #695

Merged
merged 77 commits into from
May 23, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
8673f67
feat: add fabric cicd initial single tech sample
devlace Mar 20, 2024
05b593f
feat: add folder structures
devlace Mar 20, 2024
0c6852d
Initial bootstrap changes
promisinganuj Mar 21, 2024
6cfa20b
Adding code to handle pipelines
promisinganuj Mar 21, 2024
c703465
Merge pull request #679 from Azure-Samples/promisinganuj/675-add-boot…
promisinganuj Mar 21, 2024
126fb5d
Bug fixes in bootstrap
promisinganuj Mar 21, 2024
8cbbb8b
Updated the limitations
promisinganuj Mar 21, 2024
96b7168
Adding CD release code and documentation
ydaponte Mar 21, 2024
9e738d5
Starting to update README.md
promisinganuj Mar 21, 2024
eb62eb8
Update single_tech_samples/fabric/fabric_ci_cd/README.md
ydaponte Mar 21, 2024
1035e3c
Update single_tech_samples/fabric/fabric_ci_cd/README.md
ydaponte Mar 21, 2024
f9b7547
Merge pull request #681 from Azure-Samples/ydaponte/678-add-cd-releas…
ydaponte Mar 21, 2024
550d429
Few more fixes
promisinganuj Mar 21, 2024
883991d
Merge branch 'feat/fabric_cicd' into promisinganuj/675-add-bootstrap
promisinganuj Mar 21, 2024
0741a9e
Updating README.md
promisinganuj Mar 22, 2024
fb1cea4
Updating README.md
promisinganuj Mar 22, 2024
f2511f4
Updating README.md
promisinganuj Mar 22, 2024
b969bde
Updating README.md
promisinganuj Mar 22, 2024
930cade
Updating README.md
promisinganuj Mar 22, 2024
2f13590
Updating README.md
promisinganuj Mar 22, 2024
18cf967
Merge pull request #680 from Azure-Samples/promisinganuj/675-add-boot…
promisinganuj Mar 22, 2024
42e8610
Adding architecture diagram
promisinganuj Mar 22, 2024
e3941aa
Adding architecture diagram
promisinganuj Mar 22, 2024
4f11620
Adding architecture diagram
promisinganuj Mar 22, 2024
bad6e9d
Adding architecture diagram
promisinganuj Mar 22, 2024
52aa43f
Merge pull request #682 from Azure-Samples/promisinganuj/675-add-boot…
promisinganuj Mar 22, 2024
8d9d0a1
Adding timeout, final modifications to the ReadMe
ydaponte Mar 22, 2024
202847e
Merge pull request #683 from Azure-Samples/ydaponte/678-adding-timeout
ydaponte Mar 22, 2024
dbd73f4
minor edits and fixed typos
sreedhar-guda Mar 22, 2024
ae67bac
Minor updates/typos.
sreedhar-guda Mar 22, 2024
b8a5c5c
Update README.md
jose-perales Mar 22, 2024
610a6a0
Merge pull request #684 from Azure-Samples/sguda/fabric-cicd-data-art…
sreedhar-guda Mar 22, 2024
8537ce7
Updating architecture diagram
promisinganuj Mar 25, 2024
8ad7859
Updating architecture diagram
promisinganuj Mar 25, 2024
e9bc156
Merge pull request #686 from Azure-Samples/promisinganuj/675-add-boot…
promisinganuj Mar 25, 2024
b8f3f38
Updating Diagram
promisinganuj Mar 25, 2024
b62ec73
Merge pull request #687 from Azure-Samples/promisinganuj/675-add-boot…
promisinganuj Mar 25, 2024
fb4f080
added template file, token verification, json for curl calls for headers
sreedhar-guda Mar 25, 2024
b432fcc
Minor updates
promisinganuj Mar 26, 2024
884f318
Adding a fix
promisinganuj Mar 26, 2024
70ca905
Merge pull request #688 from Azure-Samples/sguda/fabric-ci-cd-run1
promisinganuj Mar 26, 2024
e3e79d7
Correct link to scripts
ydaponte Mar 26, 2024
e389a32
Correct capital case
ydaponte Mar 26, 2024
06f3790
Link correction, scripts folder rename
ydaponte Mar 26, 2024
c5d8f04
Merge pull request #690 from Azure-Samples/ydaponte/689-correct-link-…
ydaponte Mar 26, 2024
5318984
added code to update admin privs on workspaces and deployment pipeline
sreedhar-guda Mar 26, 2024
7c5cd12
Update .envtemplate with admin info and comments
sreedhar-guda Mar 26, 2024
a90df65
Update README.md to change the order az login commands
sreedhar-guda Mar 26, 2024
ad5ed02
Reducing number of environment variables
promisinganuj Mar 28, 2024
a60bc5d
Merge pull request #693 from Azure-Samples/anuj/692-boostrap-script-e…
promisinganuj Apr 2, 2024
a9c1da5
Adding domain creation logic
promisinganuj Apr 3, 2024
d1656af
Adding domain creation logic
promisinganuj Apr 3, 2024
c7f5945
Adding domain creation logic
promisinganuj Apr 3, 2024
7663762
Adding domain creation logic
promisinganuj Apr 3, 2024
9846793
Adding domain creation logic
promisinganuj Apr 3, 2024
4cbc724
Adding domain creation options
promisinganuj Apr 4, 2024
6399c35
Adding domain creation options
promisinganuj Apr 4, 2024
13a6f57
Merge pull request #694 from Azure-Samples/anuj/692-boostrap-script-e…
promisinganuj Apr 4, 2024
b8d78bb
Merge branch 'feat/fabric_cicd' of https://github.com/Azure-Samples/m…
promisinganuj Apr 4, 2024
ccf7abf
Updating README.md
promisinganuj Apr 4, 2024
2a11a98
Updating README.md
promisinganuj Apr 4, 2024
bb45649
Updating README.md
promisinganuj Apr 4, 2024
194d435
Updating README.md
promisinganuj Apr 4, 2024
4202fa4
Updating README.md
promisinganuj Apr 4, 2024
9ad0add
Adding functionality to add admins
promisinganuj Apr 5, 2024
46b9809
Fixing downstream linting issues
promisinganuj Apr 7, 2024
85332b6
Fixing downstream linting issues
promisinganuj Apr 7, 2024
1afcdb4
Merge pull request #691 from Azure-Samples/sguda/add-ownership-code
promisinganuj Apr 8, 2024
9642a2f
Adding new environment vairable for adding existing capacity name
promisinganuj Apr 17, 2024
fbcb5ce
Merge pull request #700 from Azure-Samples/anuj/adding-existing-capac…
promisinganuj Apr 18, 2024
d02c71e
Updating README.md and architecture diagram
promisinganuj May 20, 2024
7e3b1f1
fixing linting error
promisinganuj May 20, 2024
f37f68e
Incorporating review comments + adding utility script to upload file …
promisinganuj May 22, 2024
c4353b2
Updating README.md
promisinganuj May 23, 2024
bd4074a
Updated to README.md
promisinganuj May 23, 2024
62f97f4
Updated to README.md
promisinganuj May 23, 2024
51d2493
Updating pipleline definitions to use correct variable
promisinganuj May 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions single_tech_samples/fabric/fabric_ci_cd/.envtemplate
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ AZURE_SUBSCRIPTION_ID=""
AZURE_LOCATION=""
# The email address of the Fabric capacity admin. This should be from the same tenant where capacity is being created.
CAPACITY_ADMIN_EMAIL=""
# The name of the existing Fabric capacity. This name is required if you want to use an existing capacity instead of creating a new one.
EXISTING_CAPACITY_NAME=""
# The name of the Fabric project. This name is used for naming the Fabric resources.
FABRIC_PROJECT_NAME=""
# The name of the Fabric domain. It can be an existing domain.
Expand Down
5 changes: 3 additions & 2 deletions single_tech_samples/fabric/fabric_ci_cd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ Here are the steps to use the bootstrap script:
AZURE_SUBSCRIPTION_ID='Azure Subscription Id'
AZURE_LOCATION='The location where the Azure resources will be created'
CAPACITY_ADMIN_EMAIL='The email address of the Fabric capacity admin. This should be from the same tenant where capacity is being created.'
EXISTING_CAPACITY_NAME="The name of the existing Fabric capacity. This name is required if you want to use an existing capacity instead of creating a new one."
FABRIC_PROJECT_NAME='The name of the Fabric project. This name is used for naming the Fabric resources.'
FABRIC_DOMAIN_NAME="The name of the Fabric domain. It can be an existing domain."
FABRIC_SUBDOMAIN_NAME="The name of the Fabric subdomain. It can be an existing subdomain."
Expand All @@ -80,7 +81,7 @@ Here are the steps to use the bootstrap script:

| Flag | Description | Default value | Required environment variables |
| :---------------------------------- | :-------------------------------------------------------------------- | :------------ | :----------------------------------------------------------------------------------------------------- |
| deploy_azure_resources | Flag to deploy Azure resources. | false | `AZURE_SUBSCRIPTION_ID`</br>`AZURE_LOCATION`</br>`CAPACITY_ADMIN_EMAIL` |
| deploy_azure_resources | Flag to deploy Azure resources. | false | If set to false:</br>`EXISTING_CAPACITY_NAME`</br>If set to false:</br>`AZURE_SUBSCRIPTION_ID`</br>`AZURE_LOCATION`</br>`CAPACITY_ADMIN_EMAIL` |
| create_workspaces | Flag to create new Fabric workspaces. | true | `FABRIC_PROJECT_NAME` |
| setup_deployment_pipeline | Flag to create a deployment pipeline and assign workspaces to stages. | true | `FABRIC_PROJECT_NAME` |
| create_default_lakehouse | Flag to create a default Lakehouse. | true | - |
Expand Down Expand Up @@ -188,7 +189,7 @@ The bootstrap script is designed to automate the creation of initial Fabric work

Here is a summary of the steps that the script performs:

- Creates an Azure resource group and a Fabric capacity. If the capacity already exists, the script fetches the Id based on the capacity name.
- Creates an Azure resource group and a Fabric capacity. For using an existing capacity, set `deploy_azure_resources` to `false` and provide the existing capacity name using `EXISTING_CAPACITY_NAME` environment variable.
- Creates three Fabric workspaces for development (DEV), user acceptance testing (UAT), and production (PRD). If the workspaces already exist, the script fetch the corresponding workspace Ids and writes a warning message to validate that the existing workspaces are indeed connected to the intended capacity.
- Creates a deployment pipeline and assigns the workspaces to the stages. If any pipeline stage is already associated with a different workspace, it reassigns the new workspaces to the stage.
- Creates a Fabric Lakehouse `lh_main`.
Expand Down
33 changes: 24 additions & 9 deletions single_tech_samples/fabric/fabric_ci_cd/bootstrap.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ fabric_bearer_token="$FABRIC_BEARER_TOKEN"
fabric_project_name="$FABRIC_PROJECT_NAME"
location="$AZURE_LOCATION"
capacity_admin_email="$CAPACITY_ADMIN_EMAIL"
existing_capacity_name="$EXISTING_CAPACITY_NAME"
## APIs Endpoints
fabric_api_endpoint="https://api.fabric.microsoft.com/v1"
deployment_api_endpoint="$fabric_api_endpoint.0/myorg/pipelines"
Expand Down Expand Up @@ -38,9 +39,9 @@ trigger_notebook_execution="true"
trigger_pipeline_execution="true"
should_disconnect="false"
connect_to_git="true"
create_domain_and_attach_workspaces="false"
add_workspace_admins="false"
add_pipeline_admins="false"
create_domain_and_attach_workspaces="true"
add_workspace_admins="true"
add_pipeline_admins="true"

# Derived variables (change as per the project requirements)
onelake_name="onelake"
Expand Down Expand Up @@ -72,10 +73,13 @@ function create_resource_group () {

function deploy_azure_resources() {
resource_group_name="$1"
azure_location="$2"
capacity_name="$3"
capacity_admin_email="$4"
arm_output=$(az deployment group create \
--resource-group "$resource_group_name" \
--template-file infra/main.bicep \
--parameters location="$AZURE_LOCATION" capacityName="$fabric_capacity_name" adminEmail="$capacity_admin_email" \
--parameters location="$azure_location" capacityName="$capacity_name" adminEmail="$capacity_admin_email" \
--output json)

if [[ -z $arm_output ]]; then
Expand All @@ -87,8 +91,9 @@ function deploy_azure_resources() {
}

function get_capacity_id() {
capacity_name="$1"
response=$(curl -s -X GET -H "Authorization: Bearer $fabric_bearer_token" "$fabric_api_endpoint/capacities")
capacity_id=$(echo "${response}" | jq -r --arg var "$fabric_capacity_name" '.value[] | select(.displayName == $var) | .id')
capacity_id=$(echo "${response}" | jq -r --arg var "$capacity_name" '.value[] | select(.displayName == $var) | .id')
echo "$capacity_id"
}

Expand Down Expand Up @@ -580,16 +585,26 @@ echo "[I] ############ Azure Resource Deployment ############"
if [[ "$deploy_azure_resources" = "true" ]]; then
echo "[I] Creating resource group '$resource_group_name'"
create_resource_group "$resource_group_name"

echo "[I] Deploying Azure resources to resource group '$resource_group_name'"
deploy_azure_resources "$resource_group_name"
deploy_azure_resources "$resource_group_name" "$location" "$fabric_capacity_name" "$capacity_admin_email"
else
echo "[I] Variable 'deploy_azure_resources' set to $deploy_azure_resources, skipping Azure resource deployment."
if [[ -n "$existing_capacity_name" ]]; then
echo "[I] Variable 'deploy_azure_resources' set to $deploy_azure_resources, using existing capacity '$existing_capacity_name'."
# As an existing capacity is being used, the "fabric_capacity_name" variable is overridden with the name of the existing capacity passed.
fabric_capacity_name="$existing_capacity_name"
else
echo "[E] As variable 'deploy_azure_resources' is set to $deploy_azure_resources, you need to set 'EXISTING_CAPACITY_NAME' environment variable."
exit 1
fi
fi

echo "[I] ############ Workspace Creation ############"
if [[ "$create_workspaces" = "true" ]]; then
capacity_id=$(get_capacity_id)
capacity_id=$(get_capacity_id "$fabric_capacity_name")
if [[ -z "$capacity_id" ]]; then
echo "[E] Capacity '$fabric_capacity_name' not found. Please verify the capacity name passed."
exit 1
fi
echo "[I] Fabric capacity is '$fabric_capacity_name' ($capacity_id)"

for ((i=0; i<${#workspace_names[@]}; i++)); do
Expand Down
Loading