Job Queue Daemon

Symfony console application which is used inside an ECS task and wraps Docker runner library.

Development

Prerequisites:

• configured az, aws and gcloud CLI tools (run az login, aws configure --profile Keboola-CI-Platform-Services-Team-AWSAdministratorAccess, gcloud auth)
  • Activate PIM AAD App Administrator in https://portal.azure.com/#view/Microsoft_Azure_PIMCommon/ActivationMenuBlade/~/aadgroup
  • You must have Contributor and User Access Administrator role assignet to Keboola DEV Platform Services Team subscription in Azure cloud
  • You must have access to Keboola-Dev-Platform-Services-AWSAdministratorAccess profile in AWS
  • You must have access to kbc-dev-platform-services folder in GCP
• installed terraform (https://www.terraform.io) and jq (https://stedolan.github.io/jq) to setup local env
• installed docker to run & develop the app

TL;DR:

export AWS_PROFILE=Keboola-Dev-Platform-Services-AWSAdministratorAccess
export NAME_PREFIX= # your name/nickname to make your resource unique & recognizable

cat <<EOF > ./provisioning/local/terraform.tfvars
name_prefix = "${NAME_PREFIX}"
EOF

cat <<EOF > .env.local
TEST_STORAGE_API_TOKEN= # regular token for your Keboola project
TEST_STORAGE_API_TOKEN_MASTER= # master token for your Keboola project
EOF

az acr login -n keboolapes

terraform -chdir=./provisioning/local init -backend-config="key=job-runner/${NAME_PREFIX}.tfstate"
terraform -chdir=./provisioning/local apply
./provisioning/local/update-env.sh aws # or azure

docker compose run --rm dev composer install
docker compose run --rm dev composer ci

Using Docker

Project has Docker development environment setup, so you don't need to install anything on your local computer, except the Docker & Docker Compose.

To run PHP scripts, use the dev service:

docker compose run --rm dev composer install   # install dependencies using Composer 
docker compose run --rm dev composer phpunit   # run Phpunit as a Composer script
docker compose run --rm dev vendor/bin/phpunit # run Phpunit standalone
docker compose run --rm dev bin/console        # run Symfony console commands

To run local tests, use ci service. This will validate composer files and execute phpcs, phpstan and phpunit tests.

docker compose run --rm ci

ENV & Configuration

For local development, we follow Symfony best practices as described in docs and use .env file:

• .env is versioned, should contain sane defaults to run the service out of the box locally
• .env.local is not versioned, can be created to override any ENV variable locally
• .env.test is versioned, should contain anything extra is needed for test environment

These are used for local development only and are not included in final Docker images, used to run the app in production. Instead, we put an empty .env.local.php file into Docker, disabling the .env functionality and all configuration must be provided using regular environment variables (like -e flag of Docker).

License

MIT licensed, see LICENSE file.