online-service

Requirements

• A MongoDB database or API compatible (Azure Cosmos DB)
• Azure service bus
• Kubernetes

Installation

git clone https://github.com/webhintio/online-service.git
cd online-service
npm install
npm run build

Configuration

You need to configure at least the following env variables in order to run the service:

• database: The connection string for the database
• queue: The connection string for Azure service bus
• port: The port where the job manager will be listening

Quick start user guide

Run the Job manager

npm run online-service -- --microservice job-manager

Run the Config manager

npm run online-service -- --microservice config-manager --name new-config-name --file path/to/you/config-file.json --cache 120 --run 120

NOTE: use npm run online-service -- --help to get more information about what each argument means

Run the Worker service

npm run online-service -- --microservice worker

Run the Sync service

npm run online-service -- --microservice sync

Run everything at the same time (except the config manager)

npm run online-service -- --microservice all

NOTE: You need to set up the following environment variables before continue: NODE_ENV=production, database=YourConnectionStringToTheDatabase and queue=YourConnectionStringToServiceBus

Kubernetes

Local environment

If you want to run the online-service in you local machine, you just need to run:

kubectl apply -f kubernetes.yml

NOTE:

Kubernetes needs to be installed in your computer.

We are assuming that you are in the folder compose before running kubectl. If you are in another folder, replace the file with your path to the file kubernetes.yml.

Remember you need to replace the enviroment variables values in compose/kubernetes.yml with your own values before run kubectl.

Azure environment

First of all we need to deploy kubernetes services in Azure.

To do so, first you need to go to the folder deploy inside the folder scripts.

cd scripts/deploy

Make the file deploy.sh executable.

chmod +x deploy.sh

Run deploy.sh. If you don't want to pass the parameters, the script will ask you later for the information needed.

./deploy.sh -i <subscriptionName> -g <resourceGroupName> -l <resourceGroupLocation> -k <sshPublicKey> -r <containerRegistryName>"

The deployment script will create all the necessary to run the online scanner:

1. An Azure kubernetes service.
2. An Azure Service bus.
3. A Virtual machine with linux to install nginx.
4. If needed, a replicated mongodb.
5. The peering between the linux VM and the database (the new one or an old one).

NOTE:

sshPublicKey is the path to the file containing the public ssh key.

Build images

To build all the images at the same time you need to run the script build-images.js.

To do so, first you need to go to the folder scripts.

cd scripts

Run build-images.js with the name of your repository and the image version as parameters.

node build-images.js --repository webhint --version 1

Upload the images to your Container Registry

To upload all the images at the same time you need to run the script update-images.js.

To do so, first you need to go to the folder scripts.

cd scripts

Run upload-images.js with the name of your repository and the image version as parameters.

node upload-images.js --repository webhint --version 1

NOTE: You need to be logged in into you Container Registry az acr login --name ContainerRegistryName before upload the images.

Update the configuration file

Before deploy, the configuration file needs to point to the right repository and to the current version of the images. You can do that manually or you can run the script update-config-file.js.

To do so, first you need to go to the folder scripts.

cd scripts

Run update-config-file.js with the name of your repository, the image version and the path to the configuration file as parameters.

node update-config-file.js --repository webhint --version 1 --kubernetes ../compose/kubernetes-azure.yml

NOTE:

The file path is optional, by default the value is ../compose/kubernetes-azure.yml

You also need the config from your cluster az aks get-credentials --resource-group ResourceGroupName --name ClusterName

Deploy

To deploy the online scanner in kubernetes you need to run the script deploy-kubernetes.js

To do so, first you need to go to the folder scripts.

cd scripts

Run deploy-kubernetes.js with the path to the configuration file as a parameter.

node deploy-kubernetes.js --kubernetes ../compose/kubernetes-azure.yml

NOTE:

The file path is optional, by default the value is ../compose/kubernetes-azure.yml

Build, upload, configure and deploy (all in one)

You can run all the previous steps with just one script using build-and-deploy.js

To do so, first you need to go to the folder scripts.

cd scripts

Run build-and-deploy.js with the name of your repository and the path to the configuration file as parameters.

node build-and-deploy.js --repository webhint --kubernetes ../compose/kubernetes-azure.yml

NOTE:

The version for the images will be auto calculated using your current images you have in your computer.

Deploy NGINX

To deploy NGINX, you will find a few files in scripts/deploy/nginx:

• install-nginx.sh
• configure-nginx.sh
• nginx-step1.conf
• nginx-step2.conf

Before start deploying NGINX, you need to copy these files to your NGINX machine:

Copy files into the machine where we want to deploy:

1. Copy files:

   scp *.sh *.conf nginx@your.ip:~

2. Go into the machine:

   ssh -p your.port nginx@your.ip

3. Install NGINX and Certbot:

   sudo ./install-nginx.sh

NOTE: If you get an error GPG error, look for the instructions in install-nginx.sh.

1. Configure NGINX

   sudo ./configure-nginx.sh -s <serverName> -j <jobsIpAndPort> -c <configIpAndPort>

NOTE:

jobsIPAndPort is the IP and port where the job-manager is deployed. configIpAndPort is the IP and port where the config-manager is deployed.

Configuring the scanner

After everything is deployed, the next step is to configure the scanner.

1. Authorize an user to have access to the config panel

   Access the database the way you prefer and then create a user collection if it doesn't exist already and create an item with the property name set to the user (github user) you want to give access, manually.

   {
       "name": "githubusername"
   }

2. Create the OAuth App in github.

   Go to github => Settings => Developer settings

   The Authorization callback URL will be something like: https://yourip/admin/auth/github/callback

3. Add a configuration

   Once you access to the configuration panel (https://youip/admin) the next step is to add the configuration we want to use in the scanner.

   The scanner configuration is an array of hint configuration.

   To add the scanner configuration go to the admin panel => Configuration.

Code of Conduct

This project adheres to the JS Foundation's code of conduct.

By participating in this project you agree to abide by its terms.

License

The code is available under the Apache 2.0 license.