Support Azure based locks

[ynden]

Change log

Features

• Add ability to Lock & Unlock through Azure Storage Account tables
• Support Shared Key authentication
• Support Connection string authentication
• Support Client secret authentication
• Create table if it doesn't already exist
• Normalize lock name to work with Digger's format: since Storage Account table doesn't accept # and / characters
• Provide meaningful errors to user at every step

Tests

• Add tests to cover operation done with all three authentication mechanisms
• Add test variations using a test table to test different way to authenticate
• Add a way to locally test without Azure storage account using Azurite
• Run checks before starting the tests to make sure everything's in order
• Add control to only run client secret test if using a real storage account
• Modify pipeline to install Azurite & start the services before running the tests

How does it work?

There is one mandatory environment variable the user will have to set, in order to use Azure based locks: DIGGER_AZURE_AUTH_METHOD, which can take one of the three values below:

• SHARED_KEY
• CONNECTION_STRING
• CLIENT_SECRET

Then, depending on the value of DIGGER_AZURE_AUTH_METHOD, the user will have to set other environment variables.

1. SHARED_KEY
   • DIGGER_AZURE_SA_NAME: Storage account name
   • DIGGER_AZURE_SHARED_KEY: shared key of the storage account
2. CONNECTION_STRING
   • DIGGER_AZURE_CONNECTION_STRING: connection string
3. CLIENT_SECRET
   • DIGGER_AZURE_TENANT_ID: tenant id to use
   • DIGGER_AZURE_CLIENT_ID: client id of the service principal
   • DIGGER_AZURE_CLIENT_SECRET: secret of the service principal
   • DIGGER_AZURE_SA_NAME: storage account name

Running the tests

Since we need to use a real storage account to test our features, we have two options:

1. Use a real storage account
2. Use a local storage emulator called Azurite (that will emulate Azure Storage Account, but locally)

The tests have been set so we run tests against Azurite by default, unless otherwise specified. The only disadvantage of using Azurite, is that we cannot perform a CLIENT_SECRET authentication. This is why we voluntarily Skip client secret test cases when using Azurite.

To summarize the tests that are run:

• When DIGGER_TEST_USE_REAL_SA environment variable is set to "1":
  • Run Shared Key authentication tests
  • Run Connection string authentication tests
  • Run Client secret authentication tests
• When DIGGER_TEST_USE_REAL_SA environment variable is not set, or set to "0":
  * Run Shared Key authentication tests
  * Run Connection string authentication tests

DIGGER_TEST_USE_REAL_SA is only used when testing, to skip or not Client secret test case since we are not using a real storage account.

[motatoes]

Great effort @ynden . This looks ready for merge, we just need to do some final testing with an Azure account and then go ahead with the merge

[veziak]

lgtm