Rocketchat Server Upgrade Workflow

Overview

This document describes a normal workflow for deploying updates to the Rocketchat versions running in our production, preprod and dev environments. Our servers are deployed in the vega core-infrastructures, also known as the Vega Application Platform (VAP).

• We host three instances of RC server, one in each of the VAP environments. When running commands with the kluster tool you will reference which env to run against.

'vega-dev': 'messaging.vega-dev.viasat.io'
'vega-preprod': 'messaging.vega-preprod.viasat.io'
'vega-prod': 'chatbot.vega.viasat.com'

• There are also three instances of Spinnaker where you can view, re-run pipelines and run a rollout restart.

'vega-dev': 'deck.marina.viasat.io'
'vega-preprod': 'deck-preprod.marina.viasat.io'
'vega-prod': 'deck-prod.marina.viasat.io'

• Initial deployments and configurations are outside the scope of this document but are described in this wiki

Prerequisites

• The Vega kluster tool needs to be installed and ready to use
  • NOTE: This tool was originally developed to deploy rocketchat servers in the Veranda AWS environments. We no longer host our dev, preprod, and production instances there, but instead on the VAP. However, the tool has been extended to include functionality that we still use, such as:
    • versioning and configuring rocketchat server settings
    • deploying and updating rocketchat apps (DialogFlow and Salesforce Apps used by Chatbot)
    • pulling/pushing docker images from ECR to viasat artifactory instances from where we can deploy into the VAP
• When running the kluster tool you will be asked to authenticate using alohomora, for permissions to reach the ECR instances in the Veranda AWS accounts (see Eric Rosenthal for permissions). The README in the kluster repo includes instructions for setting up alohomora on your local machine.
• You will need login permissions to Spinnaker instances so that you can deploy the new version. See the core-infrastructure team for permissions if needed.
  • NOTE: It is also possible to scale in and out using a kubectl rollout restart command, but you need access and authentication to work directly against our clusters in the VAP, and this is not the preferred method unless you are one of the core-infrastructure admins.
• You need an admin account on the Rocketchat instance you want to update.

Workflow

Our docker images are built using github actions after every merge to the develop_pwa branch.

• Docker images are programatically pushed to the widechat ECR repos with the tag latest-develop_pwa, in both the Veranda-dev and Veranda-prod AWS accounts.
• If you need to update the Rocketchat server follow steps 1 - 2
• If you need to update Rocketchat settings follow step 3
• If you need to update the RC Apps follow step 4

1. Use the Vega kluster tool on your local machine to pull -> tag -> and push these images to the viasat artifactory instances where we host images to be deployed in the VAP.
   • ./kluster pod -n vega-dev -cmd artifactory_push -tag latest-develop_pwa -ntag v3.14.0
2. Upgrade rocketchat server instances in the VAP using one of the two methods described below.
   • Update using the same tag. This will be the case if you did not increment the tag when you pushed to artifactory, i.e. the version number.
     • In the Spinnaker-UI, select the messaging application and select CLUSTERS from the left pane to view what is deployed in your namespace.
     • Click on deployment rocketchat-rocketchat to open a tab on the right. NOTE: in prod the deployment name is rocketchat-chatbot.
     • In the tab click the deployment actions dropdown and select Rolling Restart
     • That's it. A new deployment/upgrade will commence and be ready usually within a few minutes.
   • Update to deploy a new tag. This will be the case if you incremented the build number by pushing a new tag to artifactory.
     • In the Spinnaker-UI, select the messaging application and select PIPELINES from the left pane to view the deployment pipeline.
     • You will be re-running the messsaging_deploy pipeline by clicking on the Rerun pipeline with same parameters button. It is a little re-run icon to the left of the trash can icon.
     • In the GUI that appears, use the Tag dropdown to select the new tag you want to deploy. NOTE: after pushing to artifactory it can take up to 5 minutes or so for the new tag to be available in the drop down.
     • Click the run button on the bottom right and a new deployment will commence.
3. Run the version controller to apply our Rocketchat settings.
   • cd kluster/projects/veranda
   • ./kluster mongodb -name vega-preprod -namespace messaging --settings -u <rocketchat-username> -p <password> -2fa <2fa-code>
4. Upgrade or deploy the Rocketchat Apps used in the Chatbot/Salesforce integration.
   • ./kluster mongodb -name vega-preprod -namespace messaging --rc_apps -u <rocketchat-username> -p <password> -2fa <2fa-code>
   • Confirm in the rocketchat UI/Administration/Apps that the newly updated Apps are enabled; if not enable them.
   • Restart the server to guarantee all new changes take. The restart button is in the rocketchat UI/general/restart-server