Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: [cawemo-migration] Add script to migrate from Cawemo to Web Modeler #329

1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ This projects contains:
conversion
- [A Process Instance Migration Tool](./process-instance-migration) to migrate
instances of a converted process definition to Camunda Platform 8
- [A migration script](./cawemo-to-web-modeler-migration) to migrate Cawemo projects to Web Modeler
- [A curated list of community migration approaches](./migration-approaches/README.md)

## Use Snapshot Versions
Expand Down
82 changes: 82 additions & 0 deletions cawemo-to-web-modeler-migration/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Cawemo to Web Modeler Migration

This bash script migrates (copies) all projects from Cawemo to Web Modeler.

## Prerequisites

* A Cawemo enterprise license to use the Cawemo API
* API credentials for Cawemo created by an Organization Admin ([read here](https://docs.camunda.org/cawemo/1.9/reference/rest-api/overview/authentication/) how to obtain them)
* An API client for Web Modeler ([read here](https://docs.camunda.io/docs/next/apis-tools/web-modeler-api/#authentication) how to create one)
* Bash 4 or newer, or zsh (on MacOS), or any other compatible bash shell
* [jq](https://github.com/jqlang/jq/wiki/Installation) for JSON manipulation

## Setup

1. Clone the project.

2. Enter your credentials for the Cawemo API and the Web Modeler client to the top of the script into this area:

```bash
CAWEMO_USER_ID=ENTER_HERE
CAWEMO_API_KEY=ENTER_HERE

MODELER_CLIENT_ID=ENTER_HERE
MODELER_CLIENT_SECRET=ENTER_HERE
```

Make sure not to push the credentials back to GitHub.

> **Note**:
> Ideally, make sure to test the migration in a development environment. However, such an environment is not always available to you. Make sure to check the API calls in the script when you check it out from the Community Hub. In the initial version that the author prepared, the script does not alter any of the files in Cawemo, and only creates projects in Web Modeler that do not impact your users unless you manually add them as collaborators.
> **There is no warranty or liability of Camunda for using this script.**

## Usage

Run it in your shell:

**Bash**
```bash
sh ./migrate-file-cawemo-to-web-modeler.sh
```

**zsh (MacOS)**
```bash
zsh ./migrate-file-cawemo-to-web-modeler.sh
```

The script migrates each individual project and its content step-by-step. It will print the status of the migration live.

![Migration example](./migration-example.png)

In case a file could not be migrated, it prints a status message.

You also find a log file in the execution directory of the script, which lists all files and their status.

### What Is Not Included In Migration

**Element templates**

C7 Element templates (catalog files) can't be migrated automatically, since Web Modeler does only support C8 element templates. You have to migrate these files manually. [Read here](https://docs.camunda.io/docs/next/components/modeler/desktop-modeler/element-templates/defining-templates/) for more info on element templates.

**Milestones**

Milestones of your files can't be migrated. If you need older versions of your files, make sure to manually download these from the history view, or to screenshot the history log.

**Comments**

Comments can't be migrated. For important notes, add these as text annotations to your diagrams.

**Collaborators**

You have to manually add collaborators to the projects after migration. Read the next section to learn more.

## Activating Migrated Projects in Web Modeler

The migrated projects are not immediately visible in Web Modeler, since no collaborator will be assigned.
To use the migrated projects, you have to ask your organization owner to use the [super-user mode](https://docs.camunda.io/docs/next/components/modeler/web-modeler/collaboration/#super-user-mode). In this mode, the owner can see all projects, and assign relevant collaborators.

## Further Reads
* Cawemo API documentation: https://docs.camunda.org/cawemo/1.9/reference/rest-api/
* Web Modeler API documentation: https://docs.camunda.io/docs/next/apis-tools/web-modeler-api/
* Web Modeler API Swagger docs: https://modeler.cloud.camunda.io/swagger-ui/index.html
* Cawemo to Web Modeler migration guide: https://docs.camunda.io/docs/next/guides/migrating-from-cawemo/
212 changes: 212 additions & 0 deletions cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh
jonathanlukas marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also ran shellcheck on the script and there are a couple of findings which might be worth fixing:

In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 46:
for row in $(echo $PROJECTS | jq -r '.[] | @base64'); do
                  ^-------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
for row in $(echo "$PROJECTS" | jq -r '.[] | @base64'); do


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 48:
     echo ${row} | base64 --decode | jq -r ${1}
          ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                           ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
     echo "${row}" | base64 --decode | jq -r "${1}"


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 74:
    NEW_PROJECT_ID=$(jq -r .id <<< "$NEW_PROJECT")
                               ^-^ SC3011 (warning): In POSIX sh, here-strings are undefined.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 77:
    declare -A folderIds
    ^------------------^ SC3044 (warning): In POSIX sh, 'declare' is undefined.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 81:
    FILE_COUNT=$(echo $FILES | jq '. | length')
                      ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    FILE_COUNT=$(echo "$FILES" | jq '. | length')


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 86:
    for file in $(echo $FILES | jq -r '.[] | @base64'); do
                       ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    for file in $(echo "$FILES" | jq -r '.[] | @base64'); do


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 88:
            echo ${file} | base64 --decode | jq -r ${1}
                 ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                   ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
            echo "${file}" | base64 --decode | jq -r "${1}"


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 91:
        if [ $FILE_COUNT -eq $FILE_INDEX ]
             ^---------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        if [ "$FILE_COUNT" -eq $FILE_INDEX ]


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 109:
        for folder in $(echo $FILE_PATH | jq -r '.[] | @base64'); do
                             ^--------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
        for folder in $(echo "$FILE_PATH" | jq -r '.[] | @base64'); do


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 111:
                echo ${folder} | base64 --decode | jq -r ${1}
                     ^-------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                                         ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
                echo "${folder}" | base64 --decode | jq -r "${1}"


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 118:
            if [ -z "${folderIds[$OLD_FOLDER_ID]}" ]
                     ^--------------------------^ SC3054 (warning): In POSIX sh, array references are undefined.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 141:
                    NEW_FOLDER_ID=$(jq -r .id <<< "$NEW_FOLDER")
                                              ^-^ SC3011 (warning): In POSIX sh, here-strings are undefined.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 148:
                    echo "\e[1A\e[K├── ✔ Migrated folder ${GREEN}$FOLDER_NAME${NC} (ID: $OLD_FOLDER_ID)."
                         ^-- SC2028 (info): echo may not expand escape sequences. Use printf.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 152:
                PARENT_ID="\"${folderIds[$OLD_FOLDER_ID]}\""
                             ^--------------------------^ SC3054 (warning): In POSIX sh, array references are undefined.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 167:
            CONTENT=$(jq -r .content <<< "$FILE_WITH_CONTENT")
                                     ^-^ SC3011 (warning): In POSIX sh, here-strings are undefined.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 169:
            CONTENT=$(sed 's/"/\\"/g' <<< $CONTENT)
                                      ^-^ SC3011 (warning): In POSIX sh, here-strings are undefined.
                                          ^------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
            CONTENT=$(sed 's/"/\\"/g' <<< "$CONTENT")


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 171:
            CONTENT=$(tr -d '\n' <<< $CONTENT)
                                 ^-^ SC3011 (warning): In POSIX sh, here-strings are undefined.
                                     ^------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
            CONTENT=$(tr -d '\n' <<< "$CONTENT")


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 186:
            NEW_FILE=$(cat .curl_tmp)
            ^------^ SC2034 (warning): NEW_FILE appears unused. Verify use (or export if used externally).


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 193:
                echo "\e[1A\e[K$FILE_TREE_SPACES$TREE_SYMBOL ✔ Migrated file ${GREEN}$FILE_NAME${NC} (Type: $FILE_TYPE, ID: $OLD_FILE_ID)."
                     ^-- SC2028 (info): echo may not expand escape sequences. Use printf.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 199:
        let FILE_INDEX=${FILE_INDEX}+1
        ^----------------------------^ SC3039 (warning): In POSIX sh, 'let' is undefined.


In cawemo-to-web-modeler-migration/migrate-cawemo-to-web-modeler.sh line 205:
echo $''
     ^-^ SC3003 (warning): In POSIX sh, $'..' is undefined.

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- NEW_FILE appears unused. Verify u...
  https://www.shellcheck.net/wiki/SC3003 -- In POSIX sh, $'..' is undefined.
  https://www.shellcheck.net/wiki/SC3011 -- In POSIX sh, here-strings are und...

Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
#!/bin/zsh

set -eu -o pipefail

# ℹ️ Requires Bash 4 or newer, or zsh
# Recommended to run it on a test environment first

# See these docs to obtain your Cawemo API credentials: https://docs.camunda.org/cawemo/1.9/reference/rest-api/overview/authentication/
CAWEMO_USER_ID=ENTER_HERE
CAWEMO_API_KEY=ENTER_HERE

# See these docs to obtain your Web Modeler client credentials: https://docs.camunda.io/docs/next/apis-tools/web-modeler-api/#authentication
MODELER_CLIENT_ID=ENTER_HERE
MODELER_CLIENT_SECRET=ENTER_HERE

# Color definitions for terminal output formatting
GREEN='\033[0;32m'
NC='\033[0m' # No Color
B=$(tput bold)
N=$(tput sgr0)
BGREEN=$GREEN$B
NNC=$NC$N

echo "------------------------------------------------------------------------"
echo "${BGREEN}MIGRATION FROM CAWEMO TO WEB MODELER${NNC}"
echo "Files are now migrated file-by-file..."
echo "------------------------------------------------------------------------"

echo ""

echo "[$(date +'%Y-%m-%d %H:%M:%S')] Migration started" >> migration.log

HTTP_STATUS=$(curl -s -o .curl_tmp -w "%{http_code}" -u $CAWEMO_USER_ID:$CAWEMO_API_KEY "https://cawemo.com/api/v1/projects")
jonathanlukas marked this conversation as resolved.
Show resolved Hide resolved

PROJECTS=$(cat .curl_tmp)

if [ "$HTTP_STATUS" != "200" ]
then
echo "⚠ GET https://cawemo.com/api/v1/projects failed with status code $HTTP_STATUS"
echo "[$(date +'%Y-%m-%d %H:%M:%S')] ⚠ GET https://cawemo.com/api/v1/projects failed with status code $HTTP_STATUS" >> migration.log

echo "Aborted"
exit 1
fi

TOKEN=$(curl -s --header "Content-Type: application/json" --request POST --data "{\"grant_type\":\"client_credentials\", \"audience\":\"api.cloud.camunda.io\", \"client_id\":\"$MODELER_CLIENT_ID\", \"client_secret\":\"$MODELER_CLIENT_SECRET\"}" https://login.cloud.camunda.io/oauth/token | jq -r .access_token)

for row in $(echo $PROJECTS | jq -r '.[] | @base64'); do
_jq() {
echo ${row} | base64 --decode | jq -r ${1}
}

OLD_PROJECT_ID=$(_jq '.id')
PROJECT_NAME=$(_jq '.name')

echo "Migrating project ${GREEN}$PROJECT_NAME${NC} (ID: $OLD_PROJECT_ID)..."

HTTP_STATUS=$(curl -s -o .curl_tmp -w "%{http_code}" -X POST "https://modeler.cloud.camunda.io/api/beta/projects" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d "$(_jq '.')")

NEW_PROJECT=$(cat .curl_tmp)

if [ "$HTTP_STATUS" != "200" ]
then
echo "⚠ POST https://modeler.cloud.camunda.io/api/beta/projects failed with status code $HTTP_STATUS"
echo "[$(date +'%Y-%m-%d %H:%M:%S')] ⚠ POST https://modeler.cloud.camunda.io/api/beta/projects failed with status code $HTTP_STATUS" >> migration.log

echo "Aborted"
exit 1
fi

echo "[$(date +'%Y-%m-%d %H:%M:%S')] Migrated project $PROJECT_NAME (ID: $OLD_PROJECT_ID)" >> migration.log

NEW_PROJECT_ID=$(jq -r .id <<< "$NEW_PROJECT")

# maps folder IDs to original IDs to not create them twice + create file in right folder
declare -A folderIds
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not seem to work in Zsh on macOS:

./migrate-cawemo-to-web-modeler.sh: line 77: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only happens when I run the script directly with ./migrate-cawemo-to-web-modeler.sh, not with zsh migrate-cawemo-to-web-modeler.sh 🤔.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess declare does not exist in sh the shell specified by the shebang at the top of the script.

To align behaviours the first line should be changed to #!/usr/bin/env zsh

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works now when using #!/bin/zsh

@jfriedenstab please verify


FILES=$(curl -s -u $CAWEMO_USER_ID:$CAWEMO_API_KEY "https://cawemo.com/api/v1/projects/$OLD_PROJECT_ID/files")

FILE_COUNT=$(echo $FILES | jq '. | length')

echo "$FILE_COUNT files to be migrated"

FILE_INDEX=1
for file in $(echo $FILES | jq -r '.[] | @base64'); do
_file_jq() {
echo ${file} | base64 --decode | jq -r ${1}
}

if [ $FILE_COUNT -eq $FILE_INDEX ]
then
TREE_SYMBOL="└──"
else
TREE_SYMBOL="├──"
fi

OLD_FILE_ID=$(_file_jq '.id')
FILE_NAME=$(_file_jq '.name')
FILE_TYPE=$(_file_jq '.type')
FILE_PATH=$(_file_jq '.canonicalPath')

FILE_TREE_SPACES=""

# Create Folders

PARENT_ID="null"

for folder in $(echo $FILE_PATH | jq -r '.[] | @base64'); do
_folder_jq() {
echo ${folder} | base64 --decode | jq -r ${1}
}

FOLDER_NAME=$(_folder_jq '.name')
OLD_FOLDER_ID=$(_folder_jq '.id')

# skip this folder if already created
if [ -z "${folderIds[$OLD_FOLDER_ID]}" ]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got an error in this line:

./migrate-cawemo-to-web-modeler.sh: line 118: fc98fdfb-0e75: value too great for base (error token is "0e75")

Only happens when I run the script directly with ./migrate-cawemo-to-web-modeler.sh, not with zsh migrate-cawemo-to-web-modeler.sh 🤔.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works now when using #!/bin/zsh

@jfriedenstab please verify

then
# New folder
echo "├── Migrating folder ${GREEN}$FOLDER_NAME${NC} (ID: $OLD_FOLDER_ID)..."

FOLDER_REQUEST="{
\"name\": \"$FOLDER_NAME\",
\"projectId\": \"$NEW_PROJECT_ID\",
\"parentId\": $PARENT_ID
}"

HTTP_STATUS=$(curl -s -o .curl_tmp -w "%{http_code}" -X POST "https://modeler.cloud.camunda.io/api/beta/folders" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d "$FOLDER_REQUEST")

NEW_FOLDER=$(cat .curl_tmp)

if [ "$HTTP_STATUS" != "200" ]
then
echo "⚠ POST https://modeler.cloud.camunda.io/api/beta/folders failed with status code $HTTP_STATUS"
echo "[$(date +'%Y-%m-%d %H:%M:%S')] ⚠ POST https://modeler.cloud.camunda.io/api/beta/folders failed with status code $HTTP_STATUS" >> migration.log
else
NEW_FOLDER_ID=$(jq -r .id <<< "$NEW_FOLDER")

folderIds[$OLD_FOLDER_ID]=$NEW_FOLDER_ID

PARENT_ID="\"$NEW_FOLDER_ID\""

echo "[$(date +'%Y-%m-%d %H:%M:%S')] Migrated folder $FOLDER_NAME (ID: $OLD_FOLDER_ID)" >> migration.log
echo "\e[1A\e[K├── ✔ Migrated folder ${GREEN}$FOLDER_NAME${NC} (ID: $OLD_FOLDER_ID)."
fi
else
# Existing folder
PARENT_ID="\"${folderIds[$OLD_FOLDER_ID]}\""
fi

FILE_TREE_SPACES=" $FILE_TREE_SPACES"
done

if [ "$FILE_TYPE" = "TEMPLATE_GENERIC" ] || [ "$FILE_TYPE" = "TEMPLATE_SERVICE_TASK" ]
then
echo "$FILE_TREE_SPACES$TREE_SYMBOL ⚠ File ${GREEN}$FILE_NAME${NC} can't be migrated: A C7 $FILE_TYPE is not supported in C8. (ID: $OLD_FILE_ID)"
echo "[$(date +'%Y-%m-%d %H:%M:%S')] File $FILE_NAME can't be migrated: A C7 $FILE_TYPE is not supported in C8. (ID: $OLD_FILE_ID)" >> migration.log
else
echo "$FILE_TREE_SPACES$TREE_SYMBOL Migrating file ${GREEN}$FILE_NAME${NC} (Type: $FILE_TYPE, ID: $OLD_FILE_ID)..."

FILE_WITH_CONTENT=$(curl -s -u $CAWEMO_USER_ID:$CAWEMO_API_KEY "https://cawemo.com/api/v1/files/$OLD_FILE_ID")

CONTENT=$(jq -r .content <<< "$FILE_WITH_CONTENT")

CONTENT=$(sed 's/"/\\"/g' <<< $CONTENT)

CONTENT=$(tr -d '\n' <<< $CONTENT)

FILE_REQUEST="{
\"name\": \"$FILE_NAME\",
\"projectId\": \"$NEW_PROJECT_ID\",
\"parentId\": $PARENT_ID,
\"content\": \"$CONTENT\",
\"fileType\": \"$FILE_TYPE\"
}"

HTTP_STATUS=$(curl -s -o .curl_tmp -w "%{http_code}" -X POST "https://modeler.cloud.camunda.io/api/beta/files" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d "$FILE_REQUEST")

NEW_FILE=$(cat .curl_tmp)

if [ "$HTTP_STATUS" != "200" ]
then
echo "⚠ POST https://modeler.cloud.camunda.io/api/beta/files failed with status code $HTTP_STATUS"
echo "[$(date +'%Y-%m-%d %H:%M:%S')] ⚠ POST https://modeler.cloud.camunda.io/api/beta/files failed with status code $HTTP_STATUS" >> migration.log
else
echo "\e[1A\e[K$FILE_TREE_SPACES$TREE_SYMBOL ✔ Migrated file ${GREEN}$FILE_NAME${NC} (Type: $FILE_TYPE, ID: $OLD_FILE_ID)."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The character is not displayed correctly:

\e[1A\e[K├── ✔ Migrated file New BPMN Diagram (Type: BPMN, ID: 561e690c-4424-4b9e-a483-bdbec98e8176).

Probably related to this shellcheck warning (see above):
SC2028 (info): echo may not expand escape sequences. Use printf

Only happens when I run the script directly with ./migrate-cawemo-to-web-modeler.sh, not with zsh migrate-cawemo-to-web-modeler.sh 🤔.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works now when using #!/bin/zsh

@jfriedenstab please verify


echo "[$(date +'%Y-%m-%d %H:%M:%S')] Migrated file $FILE_NAME (Type: $FILE_TYPE, ID: $OLD_FILE_ID)" >> migration.log
fi
fi

let FILE_INDEX=${FILE_INDEX}+1
done
done

echo "[$(date +'%Y-%m-%d %H:%M:%S')] Migration done" >> migration.log

echo $''

echo "------------------------------------------------------------------------"
echo "${BGREEN}MIGRATION SUCCESS${NNC}"
echo "Done! Log in to Web Modeler now and enable the super-user mode (https://docs.camunda.io/docs/next/components/modeler/web-modeler/collaboration/#super-user-mode) to see the projects and assign collaborators."
echo "------------------------------------------------------------------------"
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.