Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Nexus template docs for V2 #3088

Merged
merged 5 commits into from
Jan 18, 2023
Merged

Update Nexus template docs for V2 #3088

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a439ab6
Update nexus doc to v2
jjgriff93 Jan 17, 2023
770424a
Remove upgrade notes
jjgriff93 Jan 17, 2023
722a8d4
Clarify delete options
jjgriff93 Jan 17, 2023
191867b
Fix spelling
jjgriff93 Jan 17, 2023
5a0bf71
Merge branch 'main' into jjgriff93/docs/nexus-update
tamirkamara Jan 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 15 additions & 25 deletions docs/tre-admins/setup-instructions/configuring-shared-services.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@

In general, a shared service should be installed by using the UI or API directly once its bundle has been registered on the system.

## Deploy & configure V2 Nexus service (hosted on VM)
As part of this quickstart, you will need to install the Nexus shared service, as you will be subsequently deploying Guacamole VMs that depend on public package respositories to bootstrap. Due to the TRE's Firewall blocking public access, Nexus will proxy these package repositories and make them available within the TRE for the VMs to consume.

## Deploy & configure Nexus service

!!! caution
Before deploying the V2 Nexus service, you will need workspaces of version `0.3.2` or above due to a dependency on a DNS zone link for the workspace(s) to connect to the Nexus VM.
Before deploying the Nexus service, you will need workspaces of version `0.3.2` or above due to a dependency on a DNS zone link for the workspace(s) to connect to the Nexus VM.

Before deploying the Nexus shared service, you need to make sure that it will have access to a certificate to configure serving secure proxies. By default, the Nexus service will serve proxies from `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/`, and thus it requires a certificate that validates ownership of this domain to use for SSL.

Expand Down Expand Up @@ -46,39 +48,26 @@ You can optionally go to the Nexus web interface by visiting `https://nexus-{TRE

Just bear in mind that if this service is redeployed any changes made in the Nexus UI won't be persisted. If you wish to permanently add new repositories or alter existing ones, modify the JSON files within the `./nexus_repos_config` directory and redeploy.

### Migrate from an existing V1 Nexus service (hosted on App Service)

Once you've created the new V2 (VM-based) Nexus service by following the previous section, you can migrate from the V1 Nexus service by following these steps:

1. Identify any existing Guacamole user resources that are using the old proxy URL (`https://nexus-{TRE_ID}.azurewebsites.net/`). These will be any VMs with bundle versions < `0.3.2`.

1. These will need to be either **re-deployed** with the new template versions `0.3.2` or later and specifying an additional template parameter `"nexus_version"` with the value of `"V2"`, or manually have their proxy URLs updated by remoting into the VMs and updating the various configuration files of required package managers with the new URL (`https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/`).

1. For example, pip will need the `index`, `index-url` and `trusted-host` values in the global `pip.conf` file to be modified to use the new URL.

2. Once you've confirmed there are no dependencies on the old Nexus shared service, you can delete it using the API.
You can view further information on the Nexus shared service [here](../../tre-templates/shared-services/nexus.md).

### Upgrade notes
## (Optional) Install Gitea and configure repositories

The new V2 Nexus shared service can be located in the `./templates/shared_services/sonatype-nexus-vm` directory, with the bundle name `tre-shared-service-sonatype-nexus`, which is now hosted using a VM to enable additional configuration required for proxying certain repositories.
Gitea allows you to mirror Git repositories to make them available within the TRE. This is optional as part of this quickstart.

This has been created as a separate service as the domain name exposed for proxies will be different to the one used by the original Nexus service and thus will break any user resources configured with the old proxy URL.

The original Nexus service that runs on App Service (located in `./templates/shared_services/sonatype-nexus`) has the bundle name `tre-shared-service-nexus` so can co-exist with the new VM-based shared service to enable smoother upgrading of existing resources.

## Renewing certificates for Nexus

The Nexus V2 service checks Keyvault regularly for the latest certificate matching the name you passed on deploy (`nexus-ssl` by default).
Note : This is a Gitea *shared service* which will be accessible from all workspaces intended for mirroring external Git repositories. A Gitea *workspace service* can also be deployed per workspace to enable Gitea to be used within a specific workspace.

When approaching expiry, you can either provide an updated certificate if you brought your own, or if you used the certs shared service to generate one, just call the `renew` custom action on that service. This will generate a new certificate and persist it to the Keyvault.
To install Gitea, first register the template:

## Configure Gitea repositories
```cmd
make shared_service_bundle BUNDLE=gitea
```

Note : This is a Gitea *shared service* which will be accessible from all workspaces intended for mirroring external Git repositories. A Gitea *workspace service* can also be deployed per workspace to enable Gitea to be used within a specific workspace.
You can then install via the TRE UI in the same way you did for the Nexus bundle.

By default, this Gitea instance does not have any repositories configured. You can add repositories to Gitea either by using the command line or by using the Gitea web interface.

### Command Line

Make sure you run the following commands using git bash and set your current directory as C:/AzureTRE.

1. On the jumbox, run:
Expand All @@ -99,6 +88,7 @@ From a virtual machine within a workspace:
- Command line: ```git clone https://gitea-<TRE_ID>.azurewebsites.net/giteaadmin/<NameOfrepository>```
- Gitea Web Interface: ```https://gitea-<TRE_ID>.azurewebsites.net/```

More info on the Gitea shared service is available [here](../../tre-templates/shared-services/gitea.md).

## Next steps

Expand Down
28 changes: 27 additions & 1 deletion docs/tre-templates/shared-services/gitea.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,38 @@ Documentation on Gitea can be found here: [https://docs.gitea.io/](https://docs.

## Deploy

To deploy this shared service you should use the UI (or the API) to issue a request. If you don't see the option available for this specifc template make sure it has been built, published and registered by the TRE Admin.
To deploy this shared service you should use the UI (or the API) to issue a request. If you don't see the option available for this specific template make sure it has been built, published and registered by the TRE Admin.

## Getting Started

Connect to the Gitea admin console `https://yourtreuri/gitea/` with the `giteaadmin` user. You can find the password in keyvault as `gitea password`.

## Configuring repositories

By default, this Gitea instance does not have any repositories configured. You can add repositories to Gitea either by using the command line or by using the Gitea web interface.

### Command Line

Make sure you run the following commands using git bash and set your current directory as C:/AzureTRE.

1. On the jumbox, run:
```./templates/workspace_services/gitea/gitea_migrate_repo.sh -t <tre_id> -g <URL_of_github_repo_to_migrate>```
1. If you have issues with token or token doesn't work, you can reset the token by setting it's value to null in Key Vault:
```az keyvault secret set --name gitea-<tre-id>-admin-token --vault-name kv-<tre-id> --value null```

### Gitea Web Interface

1. on the jumbox, open Edge and go to:
```https://gitea-<TRE_ID>.azurewebsites.net/```
1. Authenticate yourself using username ```giteaadmin``` and the secret ```<gitea-TRE_ID-administrator-password>``` stored in the keyvault,
1. Add the repository of your choice

### Verify can access the mirrored repository

From a virtual machine within a workspace:
- Command line: ```git clone https://gitea-<TRE_ID>.azurewebsites.net/giteaadmin/<NameOfrepository>```
- Gitea Web Interface: ```https://gitea-<TRE_ID>.azurewebsites.net/```

## Network requirements

Gitea needs to be able to access the following resource outside the Azure TRE VNET via explicitly allowed [Service Tags](https://docs.microsoft.com/en-us/azure/virtual-network/service-tags-overview) or URLs.
Expand Down
94 changes: 75 additions & 19 deletions docs/tre-templates/shared-services/nexus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,60 @@ Documentation on Nexus can be found here: [https://help.sonatype.com/repomanager

## Deploy

To deploy this service use the UI or API directly and choose the nexus template.
!!! caution
Before deploying the Nexus service, you will need workspaces of version `0.3.2` or above due to a dependency on a DNS zone link for the workspace(s) to connect to the Nexus VM.

Nexus will be deployed as part of the main TRE terraform deployment. A configuration script needs to be run once the deployment is done. The script will:
Before deploying the Nexus shared service, you need to make sure that it will have access to a certificate to configure serving secure proxies. By default, the Nexus service will serve proxies from `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/`, and thus it requires a certificate that validates ownership of this domain to use for SSL.

1. Fetch the Nexus generated password from storage account.
2. Reset the default password and set a new one.
3. Store the new password in Key Vault under `nexus-<TRE_ID>-admin-password`
4. Create an anonymous default PyPI proxy repository
You can use the Certs Shared Service to set one up by following these steps:

1. Run the below command in your terminal to build, publish and register the certs bundle:

```cmd
make shared_service_bundle BUNDLE=certs
```

2. Navigate to the TRE UI, click on Shared Services in the navigation menu and click *Create new*.

3. Select the Certs template, then fill in the required details. *Domain prefix* should be set to `nexus` and *Cert name* should be `nexus-ssl`.

!!! caution
If you have KeyVault Purge Protection enabled and are re-deploying your environment using the same `cert_name`, you may encounter this: `Status=409 Code=\"Conflict\" Message=\"Certificate nexus-ssl is currently in a deleted but recoverable state`. You need to either manually recover the certificate or purge it before redeploying.

Once deployed, the certs service will use Letsencrypt to generate a certificate for the specified domain prefix followed by `-{TRE_ID}.{LOCATION}.cloudapp.azure.com`, so in our case, having entered `nexus`, this will be `nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com`, which will be the public domain for our Nexus service.

You can verify whether this has been successful by navigating to your core keyvault (`kv-{TRE_ID}`) and looking for a certificate called `nexus-ssl` (or whatever you called it).

After verifying the certificate has been generated, you can deploy Nexus:

1. Run the below command in your terminal to build, publish and register the Nexus shared service bundle:

```cmd
make shared_service_bundle BUNDLE=sonatype-nexus-vm
```

1. Navigate back to the TRE UI, and click *Create new* again within the Shared Services page.

1. Select the Nexus template then fill in the required details. The *SSL certificate name* should default to `nexus-ssl`, so there's no need to change it unless you gave it a different name in the previous step.

This will deploy the infrastructure required for Nexus, then start the service and configure it with the repository configurations located in the `./templates/shared_services/sonatype-nexus-vm/scripts/nexus_repos_config` folder. It will also set up HTTPS using the certificate you generated in the previous section, so proxies can be served at `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com`.

## Setup and usage

1. A TRE Administrator can access Nexus though the admin jumpbox provisioned as part of the TRE deployment. The username is `adminuser` and the password is located in the KeyVault under `vm-<tre-id>-jumpbox-password`
2. A researcher can access Nexus from within the workspace by using the internal Nexus URL of: [https://nexus-<TRE_ID>.azurewebsites.net/](https://nexus-<TRE_ID>.azurewebsites.net/)
3. To fetch Python packages from the PyPI proxy, a researcher can use pip install while specifying the proxy server:
2. A researcher can access Nexus from within the workspace by using the internal Nexus URL of `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com`
3. To fetch Python packages from the PyPI proxy, a researcher can use `pip install` while specifying the proxy server:

```bash
pip install packagename --index-url https://nexus-<TRE_ID>.azurewebsites.net/repository/apt-pypi/simple
pip install packagename --index-url https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/apt-pypi/simple
```

!!! info
In the built-in Linux and Windows Guacamole VM bundles, PyPI and several other package managers are already configured to use the Nexus proxy by default, so manually specifying in the install commands isn't neccesary.

## Network requirements

Nexus Shared Service requires access to resources outside of the Azure TRE VNET. These are set as part of the firewall provisioning pipeline via explicit allow on [Service Tags](https://docs.microsoft.com/en-us/azure/virtual-network/service-tags-overview) or URLs.
Notice that since Nexus Shared Service is running on an App Service, the outgoing exceptions are made for the calls coming out of the Web App Subnet.

| Service Tag / Destination | Justification |
| --- | --- |
Expand All @@ -46,12 +77,37 @@ Notice that since Nexus Shared Service is running on an App Service, the outgoin

| Name | Type | Source URI | Nexus URI | Usage |
| --- | --- | --- | --- | --- |
| PiPy | PiPy | [https://pypi.org/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/pypi/` | Allow use of pip commands. |
| Apt PiPy | Apt | [https://pypi.org/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/apt-pypi/` | Install pip via apt on Linux systems. |
| Conda | conda | [https://repo.anaconda.com/pkgs/main/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/conda/` | Configure conda to have access to default conda packages. |
| Conda-Forge | conda | [https://conda.anaconda.org/conda-forge/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/conda-forge/` | Configure conda to have access to conda-forge packages. |
| Docker | apt | [https://download.docker.com/linux/ubuntu/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/docker/` | Install Docker via apt on Linux systems.|
| Docker GPG | raw | [https://download.docker.com/linux/ubuntu/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/docker-public-key/` | Provide public key to sign apt source for above Docker apt. |
| Docker Hub | docker | [https://registry-1.docker.io] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/docker-hub/` | Provide docker access to public images repo. |
| Ubuntu Packages | apt | [http://archive.ubuntu.com/ubuntu/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/ubuntu/` | Provide access to Ubuntu apt packages on Ubuntu systems. |
| Ubuntu Security Packages | apt | [http://security.ubuntu.com/ubuntu/] | `https://nexus-<TRE_ID>.azurewebsites.net/repository/ubuntu-security/` | Provide access to Ubuntu Security apt packages on Ubuntu systems. |
| PyPI | PyPI | [https://pypi.org/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/pypi/` | Allow use of pip commands. |
| Conda | conda | [https://repo.anaconda.com/pkgs] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/conda-repo/` | Configure conda to have access to default conda packages. |
| Conda Mirror | conda | [https://conda.anaconda.org] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/conda-mirror/` | Configure conda to have access to conda mirror packages. |
| Docker | apt | [https://download.docker.com/linux/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/docker/` | Install Docker via apt on Linux systems. |
| Docker GPG | raw | [https://download.docker.com/linux/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/docker-public-key/` | Provide public key to sign apt source for above Docker apt. |
| Docker Hub | docker | [https://registry-1.docker.io] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/docker-hub/` | Provide docker access to public images repo. |
| Ubuntu Packages | apt | [http://archive.ubuntu.com/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/ubuntu/` | Provide access to Ubuntu apt packages on Ubuntu systems. |
| Ubuntu Security Packages | apt | [http://security.ubuntu.com/ubuntu/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/ubuntu-security/` | Provide access to Ubuntu Security apt packages on Ubuntu systems. |
| Almalinux | yum | [https://repo.almalinux.org] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/almalinux` | Install Almalinux packages |
| R-Proxy | r | [https://cran.r-project.org/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/r-proxy` | Provide access to CRAN packages for R |
| Fedora Project | yum | [https://download-ib01.fedoraproject.org] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/fedoraproject` | Install Fedora Project Linux packages |
| Microsoft Apt | apt | [https://packages.microsoft.com] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-apt` | Provide access to Microsoft Apt packages |
| Microsoft Keys | raw | [https://packages.microsoft.com/keys/] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-keys` | Provide access to Microsoft keys |
| Microsoft Yum | yum | [https://packages.microsoft.com/yumrepos] | `https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/repository/microsoft-yum` | Provide access to Microsoft Yum packages |

### Migrate from an existing V1 Nexus service (hosted on App Service)

If you still have an existing Nexus installation based on App Service (from the original V1 bundle), you can migrate to the VM-based Nexus service by following these steps:

1. Install the new Nexus service alongside your old installation using the steps from earlier in this document.

1. Identify any existing Guacamole user resources that are using the old proxy URL (`https://nexus-{TRE_ID}.azurewebsites.net/`). These will be any VMs with bundle versions < `0.3.2` that haven't been manually updated.

1. These will need to be either **re-deployed** with the new template versions `0.3.2` or later and specifying an additional template parameter `"nexus_version"` with the value of `"V2"`, or manually have their proxy URLs updated by remoting into the VMs and updating the various configuration files of required package managers with the new URL (`https://nexus-{TRE_ID}.{LOCATION}.cloudapp.azure.com/`).

1. For example, pip will need the `index`, `index-url` and `trusted-host` values in the global `pip.conf` file to be modified to use the new URL.

2. Once you've confirmed there are no dependencies on the old Nexus shared service, you can delete it using the API or UI.

## Renewing certificates for Nexus

The Nexus service checks Keyvault regularly for the latest certificate matching the name you passed on deploy (`nexus-ssl` by default).

When approaching expiry, you can either provide an updated certificate into the TRE core KeyVault (with the name you specified when installing Nexus) if you brought your own, or if you used the certs shared service to generate one, just call the `renew` custom action on that service. This will generate a new certificate and persist it to the Keyvault, replacing the expired one.