RAPyDo 3.0 (#382)

🌟 New functionalities

• Added docker builds (moved from build-templates)
• Added support for python 3.12
• Added python 3.10 legacy backend
• Replaced Flask-Migrate commands with Alembic alternatives
• Password expiration can now be configured via PASSWORD_EXPIRATION_WARNING variable
• Configured PASSWORD_EXPIRATION_WARNING to disable password expiration checks when set to 0
• Add SMTP_REPLYTO env variable
• Add custom angular controls template
• Added support for ENABLE_ANGULAR_MULTI_LANGUAGE conf
• Added async_timeout to backend builds (used by Redis instead of asyncio)
• Set CYPRESS_CACHE_FOLDER to avoid permission errors
• Automatic fallback from pip3 to pip in case of exec errors
• Extended flake8 configuration with flake8-bugbear
• Add PIP_BIN env variable
• Add pytest-rerunfailures to restart flaky tests
• Improved mypy config
• Merged static analysis tools into a single workflow

🗑️ Dropped functionalities

• Dropped build-templates submodule
• Drop legacy38 backend build
• Drop support for google-analytics
• Drop BACKEND_BUILD_MODE and PYTHON_PATH in favor of BACKEND_PYTHON_VERSION
• Remove pre-commit bumps automerge
• Drop test command and controller build
• Drop ggshield hook
• Remove import of workspace-tools and typescript plugins, now included by default with yarn 4
• Drop unrecognized document-domain Permissions-Policy header

🐛 Bug fixes and improvements

• Flask 2.2 compatibility fixes, remove FLASK_ENV, add FLASK_DEBUG
• Compose v2.14 compatibility fix: removed entrypoint and command when null
• Compatibility fix for adminer (now based on ubuntu)
• Temporary added env var SQLALCHEMY_SILENCE_UBER_WARNING to silence SQLAlchemy 2.0 uber warnings
• Added SQLALCHEMY_WARN_20 env var to to raise sqlalchemy RemovedIn20Warning deprecation warnings

🚀 Dependencies Upgrades

• Bump docker buildx to v0.13.1
• Bump docker compose to v2.24.6
• Bump docker registry to v2.8.3
• Bump neo4j to v4.4.32
• Bump nginx to v1.25.5
• Bump postgres to v15.6
• Bump Node.js to v20.9.0
• Bump rabbitmq to v3.13.1
• Bump redis to v7.2.4
• Bump python backend to 3.11.3
• Bump swagger-ui to v5.16.2
• Bump fail2ban to v1.0.2
• Bump ftp image to buster
• Bump certbot to 2.10.0
• Bump gunicorn to 22.0.0
• Bump angular/cli to 15.2.10
• Bump ajv to 8.12.0
• Bump ts-json-schema-generator to 1.5.1
• Bump click to 8.1.7
• Bump cypress to 13.8.0
• Bump deepmerge to 4.3.1
• Bump dotenv to 16.4.5
• Bump python-on-whales to 0.70.1
• Bump typer to 0.9.0
• Bump GitPython to 3.1.43
• Bump pydantic to 2.7.0
• Bump requests to 2.31.0
• Bump gevent to 24.2.1
• Bump lxml to 5.2.1
• Bump PyYAML to 6.0.1
• Bump pip to 24.0
• Bump setuptools to 69.2.0
• Bump wheel to 0.41.3
• Bump pytest to 8.1.1
• Bump pytest-cov to 5.0.0
• Bump pytest-rerunfailures to 14.0
• Bump pytest-sugar to 1.0.0
• Bump pytest-timeout to 2.2.0
• Bump pytest-timeout to 2.3.1
• Bump freezegun to 1.4.0
• Bump types-python-dateutil to 2.9.0.20240316
• Bump types-pytz to 2024.1.0.20240203
• Bump types-PyYAML to 6.0.12.8
• Bump types-requests to 2.31.0.20240406
• Bump actions/setup-python to v5
• Bump checkout action to v4
• Bump codeql-action action to v3
• Bump docker/login-action to v3
• Bump upload-artifact action to v4
• Bump pre-commit hooks

RAPyDo 2.4

🚀 Dependencies Upgrades

• Bump buildx to v0.9.1

• Bump compose to v2.9.0

• Bump python-on-whales to 0.54.0

• Bump click to 8.1.3

• Bump typer to 0.7.0

• Bump GitPython to 3.1.29

• Bump pip to 22.3.1

• Bump setuptools to 65.6.3

• Bump wheel to 0.38.4

• Bump pytest to 7.2.0

• Bump pytest-cov to 4.0.0

• Bump pytest-sugar to 0.9.6

• Bump mypy to 0.991

• Bump types-python-dateutil to 2.8.19.4

• Bump types-pytz to 2022.6.0.1

• Bump types-PyYAML to 6.0.12.2

• Bump types-requests to 2.28.11.5

• Bump pre-commit hooks

🌟 New functionalities

• Bump RAPyDo version to 2.4

• Added ngx-spinner type configuration (SPINNER_TYPE env variable)

• Added types-setuptools and do to mypy deps

• Enabled Renovate

• Logging driver is now configurable and defaulted to json-file in dev mode and syslog in prod mode

• Replaced tabulate with rich

• Enable Mend Bolt

• Added mypy service

• Added support to install compose and buildx on MacOS

• Authorized the installation of docker script with checksum f0914813fcbbe35f1358a994cff812d3

• Switched from setup to pyproject (added req to setuptools 64 to support editable installation)

• Added pre-commit exec to CI

• Added support for python 3.11

🗑️ Dropped functionalities

• Dropped telegram poc

• Dropped TESTING_TOTP_HASH in favour of a mocked TOTP code in testing mode

• Dropped schemathesis

• Disabled Neo4j GDS configuration

• Dropped unused DEBUG_ENDPOINTS env variable

• Dropped support for backup, restore and password commands on mariadb

• Dropped support for mariadb auth service

🐛 Bug fixes and improvements

• Cypress integration folder renamed into e2e

• Added mount of /http-api into http-api-package to let PR to force packages installation

• Redis backup compatibility fix: with redis 7 AOF changed from a single file to a folder

• Merged projects pre-commit configurations

• Enabled precommit upgrade via Renovate

• MacOS compatiblity fix, dropped editable prefix

• Bug fix to allow docker registry port relocation

• Increased bash columns used on github actions

• Added compose and buildx under renovate control

• Enabled Renovate automerge for patch deps

• Fixed pyproject.toml to include package data

• Added types optional-dependencies

• Pinned dev and stubs dependencies

• Moving depdendencies to requirements files via dynamic metadata

• Added requirements.dev.txt and requirements.types.txt to Renovate conf

• Typing fixes after the new no_implicit_optional=True default

• Enabled Renovate automerge for precommit deps

• Test fix, added exec-opts to docker json

• Fix flake8 url from gitlab to github

🛡️ Fail2ban rules

• Added Generic ADSL Router DNS Change fail2ban rule

• Updated list of permanent IP bans

RAPyDo 2.3 (#40)

🚀 Dependencies Upgrades

• Bump RAPyDo version to 2.3

• Bump python-on-whales to 0.40.0

• Bump GitPython to 3.1.27

• Bump mariadb to 10.8.2

• Bump typer to 0.4.1

• Bump click to 8.1.2

• Bump buildx to v0.8.2

• Bump compose to v2.5.0

• Bump python-version to 3.10 in CI workflows

• Precommit update

• Upgraded GA build image from ubuntu-20.04 to ubuntu-latest

🌟 New functionalities

• Enabled neo4j apoc core functions

• Enabled neo4j gds functions

• Set Redis as default broker when rabbit is not enabled

• Added backend-legacy to valid BACKEND_BUILD_MODE values

• Moved metadata from setup.py to setup.cfg

• Added project metadata to pyproject.toml

• Converted rapydo install to always work at user level

• Added ENABLE_YARN_PNP flag (disabled by default)

• Configured fail2ban persistent bans based on a custom IP blacklist

• Added a parsing of iptables --version to switch between legacy and nft

• Added a script based on AbuseIPDB APIs to verify blacklisted IPs

🗑️ Dropped functionalities

• Disabled Deadpendency workflow

• Dropped MongoDB

🐛 Bug fixes and improvements

• Bug fix to correctly run the maintenance service, due to the run behaviour port 443 was not properly mapped

• Bug fix to correctly assign service port to SwaggerUI and Adminer notification messages

• Added variables for the FTP connector

• Set redis-cli as default command on redis shell

• Added passwords expiration warnings on check

• Moved isort configuration from .isort.cfg to pyproject.toml

• Replaced Sultan with Plumbum in packages commands

• Updated PYTHON_PATH

• Authorized the installation of docker script with checksum c3a774bf0e34387a0414f225d4dd84d9

• Removed replicas if both replicas and global mode are set

• Reimplemented frontend reload to always start the frontend build in compose mode

• Bug fix to allow ACTIVATE_FAIL2BAN flag via cli

• Fail2ban configuration is now dynamically loaded based on activated services

🛡️ Fail2ban rules

• Added ColdFusion administrator access fail2ban rule

• Added Symantec Secure Web Gateway RCE rule to fail2ban configuration

• Added CVE-2022-22963 to fail2ban configuration

• Added CVE-2020-10987 to fail2ban ruleset

• Added fail2ban rule to detect Shenzhen TVT DVR/NVR/IPC attempts

• Added CVE 2022-1388 to fail2ban rules

• Added url encoded cgi-bin URLs to fail2ban rules

• Added CVE-2014-2321 to to fail2ban rules

RAPyDo 2.2 (#38)

* Bump RAPyDo version to 2.2

* Bump buildx to v0.7.1

* Bump compose to v2.2.3

* Bump python on whales to 0.36.1

* Bump mariadb to 10.7.1

* Bump mypy to 0.930

* Bump PyYAML to 6.0

* Bump GitPython to 3.1.25

* Bump click to 8.0.4

* Dropped official support for python 3.7

* Dropped frontend courtesy page

* Added tests on python 3.10

* Automatically deleted yarn.lock on rapydo init

* Added pydantic validation of project configuration

* Configuration is now typed-dict-aware

* Added a check on installed compose and buildx version

* Added CURRENT_UID and CURRENT_GID to postgres container

* Added support for custom commands

* Implemented modularization of backup, restore, tuning and password commands

* Merged together Swarm wrapper and ComposeV2 wrapper to simplify the interface

* Modularization of registry utilities

* Extended fail2ban configuration with jails against CVE-2021-44228

* Swarm mode can now be activated from the .projectrc (swarm: True)

* Increased ping timeout in celery health check from 1 (default) to 5

* Changed celery health check command to use an internal health check script

* BACKEND_URI renamed into BACKEND_URL

* Added FRONTEND_URL variable (containing a full URL, BASE_HREF and FRONTEND_PREFIX are ignored if FRONTEND_URL is set)

* Added BACKEND/FRONTEND URL variables to backend and celery containers

* Bug fix to prevent the cannot determine build priority warning with custom services

* Added YARN_ENABLE_IMMUTABLE_INSTALLS env to frontend container

* Redis is now based on the custom wrapper image

* Added UID e GID env variables to Redis container

* Set redis default user for both shell and backup commands

* Added a sleep before restarting services after backup and restore to prevent connectors errors

* Replaced <HOST> with <ADDR> on all fail2ban filters

* Experimental add of python debug flags in testing mode

* Replaced deprecated distutils.version.LooseVersion with packaging.version.Version

* Added packaging to dependencies

* Bug fix to prevent orphaned file handlers

* Bug fix to properly close sockets

* Enabled sudo on installs executed in testing mode

* Replaced deprecated distutils.core.run_setup in check_updates

* Dropped pushpin service

* Set usedns no on nginx jail

* Setting celery Execution Pool as env variable, default to prefork

* Bug fix: fail2ban maxretry was not properly set for backend jail

* Restart command is now replaced by start --force

* Bug fix: reload frontend in prod mode in case the container is missing

* Added env variable to configure celery pool mode

* Remote syslogging is no longer available due to the extreme slowness -> fail2ban is now replicated on each node in the cluster

* Password command is now placeholders-aware

* Added Sphinx and Readthedocs setup

* Added CVE-2019-16759 fail2ban rule

* Added CVE-2021-41277 fail2ban rule

* Added CVE-2021-42013 fail2ban rule

* Added rule to detect abuse of Weblogic12c T3 protocol

* Passing list of enabled Auth Roles to Cypress

* Set connection timeout (3 seconds) to backend healtchecks

* Added roles_descriptions to the main configuration

* Fixed isort configuration to prevent incompatibilities between isort and black

RAPyDo 2.1 (#37)

* Bump RAPyDo version to 2.1

* Dropped unused read_extended flag of controller_init method

* Dropped -S / --skip-services option

* Reimplemented skip services by using and underscore as naming convention for skipped services, 
e.g. _frontend to get all services expected frontend

* Added positional service argument to commands requiring targets

* Local IP is now resolved as 127.0.0.1 in development mode

* Compose-cli is now mandatory also in compose mode

* Reimplemented get_config from compose to compose v2 and introduced typed compose configs

* Fixed update command

* Rapydo logs in swarm mode is now based on docker service logs

* Added GitGuardian to pre-commit hooks

* Bug fix to verify placeholders before registry command

* Bump mariadb 10.6.4

* Bug fix to prevent empty variables to be exluded in the containers environment

* Reimplemented restart command in swarm mode to re-deploy the stack

* Added condition: none to frontend container in swarm and production mode

* Improved placeholders checks to support custom services

* Deprecated support for python 3.7

* Added PYTHONUNBUFFERED variable to backend and celery containers

* Removed obsolete compose test workflow

* Replaced dump config from compose v1 to v2

* Enabled list command in swarm mode

* Added a sleep after remove in swarm mode to let the command to be chained with a ollowing start 
command

* Remove command in swarm mode is now sync by waiting the network complete removal

* Status command in compose mode reimplemented with compose v2

* Registry command reimplemented with compose v2

* Stop command reimplemented with compose v2

* Start command reimplemented with compose v2

* Remove command reimplemented with compose v2

* Fixed start command in compose mode to prevent re-execution [to be coherent with swarm 
implementation]

* Enabled rapydo remove --all [services list] in compose mode

* Dropped services argument from restart command in compose mode to be coherent with swarm 
implementation

* Restart command reimplemented with compose v2

* Dropped deprecated interfaces options swagger, celery and mongo

* Bug fix: added verify available images before scale in swarm mode

* Reimplemented registry command as volatile (back to compose v1)

* Fixed volatile containers name to match the service

* Replaced registry command with run registry

* Replaced volatile command with run --debug service

* Replaced interfaces command with run service

* Bug fix to exclude swarm_options on run commands

* Re-added status information to list services command

* Enabled dump command in swarm mode

* Implemented --force flag on restart command

* Extracted last commit hash and saved as BUILD env variable on both backend and frontend containers

* Implemented rapydo remove registry command

* Added Size column to rapydo images output

* Added explicit message in case of no running containers with status command in compose mode

* Scale command reimplemented with compose v2

* Changed AUTH_DEFAULT_PASSWORD as for other default passwords

* Added check of password strength via zxcvbn package

* Introduced restart --force in compose mode

* Dropped obsolete mongoui variables (replaced by adminer)

* Stub of password command

* Included Flower to services supported by the create command

* Dropped logs --no-color flag

* Added password expiration check on password command

* Configured test packages as extras_requires

* Dropped compose installation script, based on official documentation the binary itself should be 
enough

* Implemented rapydo password flower

* Replaced the base faker fixture with a custom version to prevent the fixed zero-seed

* Bump redis to 6.2.5

* Bump pushpin to 1.33.1

* Bug fix in random_project_name, project names are expected to be in lower case

* Bump buildx to 0.6.3

* Bump typer to 0.4.0

* Bug fix to prevent errors on run registry when the container is in exited/stopped status

* Bug fix: added image check on run registry command

* Added tabulate package to format tables

* Implemented change of registry password

* Removed check on short passwords in production mode, replaced with the new password strenght checks

* Formatted placeholders variable with tabulate

* Added --show flag to rapydo password command

* Nodes in swarm status are now tabulated

* Added a check to prevent registry to be started in compose mode

* Added a check on missing volume paths in nfs mode

* Enabled remove --all in compose mode

* Reimplemented run command on compose v2

* Disabled service ports on volatile containers

* Added a test to verify command execution outside a valid project folder

* Bug fix: restart and remove in swarm mode were not verifying the folder validity as first step

* Introduced new coverage action

* Bump GitPython to 3.1.24

* Added optional path to upgradeable paths

* Added list of pulled commits to update output

* Updated codeql workflow

* Enabled ossar and semgrep analyses

* Added noopener rel to _blank anchor in custom brand template

* Renamed UPLOAD_PATH into DATA_PATH

* Added mode: global to nginx service and mode: host to nginx 443 exposed port. This setting prevents 
nginx to be behind the swarm proxy. Cons: nginx is no longer scalable. Pro: nginx is able to access the real client IP address, instead of the load balancer IP

* Disabled frontend health checks in production+swarm mode

* Bug fix: count resources required for the stack in case of global services

* Dropped obsolete RABBITMQ_CTL_ERL_ARGS (-proto_dist inet_tls)

* Dropped yaml.OrderedLoader leveraging ordered dicts in python3.7+

* Replaced unsafe yaml.load_all with yaml.safe_load_all

* Redirected all containers logs to syslog

* Added fail2ban service

* Restricted fail2ban in swarm mode to run on the manager node

* Added fail2ban default jails

* Bug fix to make fail2ban to work in swarm mode

* Bump compose reference version to 3.9

* Fail2ban nginx jail

* Added requested command and parameters to controller log file

* AUTH_MAX_LOGIN_ATTEMPTS is now enabled by default with 8 attempts

* Added RabbitMQ, postgres, mongodb and mysql fail2ban jails

* Dropped the constraints preventing the start command to work after the first stack deployment, now 
start can be used again to re-deploy a stack. Please note that additional stacks are aggregated in compose mode and are replacing the previous one in swarm mode

* Fixed proxy http port on the host network

* Added docker.compose.version to initial checks

* Reimplemented shell command by using python on whales instead of compose v1

* Replaced compose v1 based exec command in ssl and tuning with new docker based command

* Get container utility now only returns a container if in running, starting or ready status to 
prevent commands on exited containers

* Replaced compose v1 based exec command in backup and restore with new docker based command

* Completed rapydo images --remove command by adding execution of garbage collector and registry 
restart in order to clean the layers cache

* Dropped obsolete exec_command, get_containers_status, get_running_containers in compose v1 wrapper

* Added a parser of exit codes of container command execution based on standard chroot exit codes

* Added a test command to locally execute tests on a dedicated container

* Removed check of buildx and compose version from base checks to speed up all commands

* Reduced load time of main entrypoint by moving all import into the main function and prevent any 
eager loading

* Added semgrep template

* Deprecated --no-tty flags from shell and ssl commands. tty is now automatically detected and 
disabled when missing

* Enabled shell command in swarm mode

* Extended remove command to interfaces

* Command password extended to backend, neo4j, postgres, rabbit and mariadb

* Added pytest-sugar to test environment

* Bug fix: removed duplicated AUTH_DEFAULT_PASSWORD variable in testing projectrc template

* Bug fix check command in case of files with no blame commit

* Added stub of reload command

* Python version used in tests is now randomized

* Defaulted HEALTHCHECK_INTERVAL=1s in tests

* Simplified logging format

* Testing GA python cache

* Added wait_util utility to test suite

* Bug fix: get container for services in global mode (replaced slot with node_id)

* CELERY_BROKER renamed into CELERY_BROKER_SERVICE to prevent config clashes during service reload. 
Renamed CELERY_BACKEND into CELERY_BACKEND_SERVICE for homogeneity

* Bug fix: update command was not showing pulling commits

* Added --replica option to shell command

* Implemented --broadcast mode in shell command

* Defaulted slot in get_container to 1

* Implemented execution of commands on remote containers

* ssl reload on rabbit is now broadcasted

* Fail2ban ignoreip is now dynamically set based on the docker network

* Bug fix: get container automatic fallback to slot 0 when default slot is missing, this fixes the 
access to containers deployed in global mode

* Reload command is now broadcasted to all replicas

* Bug fix to let reload command to inspect remote containers

* Added template of CodeQL analysis workflow

* Bug fix: correct color label for global services in status output in swarm mode

* Bug fix: name of containers for global services

* Implemented frontend reload in production mode (swarm mode only, waiting for compose restart to 
complete on compose mode)

* Bug fixes in obsolete image checks

* Dropped experimental celery autoreload

* Enhanced the parser used to store changed passwords into the projectrc file

* Added image verify before restart command

* Swarm mode: added a loop to wait for containers start

* Swarm mode: support for multiple services in logs (excluding follow flag)

* Enabled DOCKER_SUBNET on every command

* Enabled interactive and streamed run commands

* Disabled tty and stream flags in detach mode of run command

* Adminer and SwaggerUI are now executed in non-detached mode by default

* Enabled prod frontend reload in compose mode

* Renamed deploy.compose into deplot.compose_legacy

* Enabled ssl and tuning command in swarm mode

* Bump docker compose to v2.1.0

* Enabled backup and restore commands in swarm mode

* Fixed compose deprecation warning: network.external.name is deprecated in favor of network.name

* Bug fix to automatically force compose engine when running volatile containers in swarm mode

* Dropped limit req for assets zone

* Bug fix to enable remote syslog in swarm multihost mode

* Fail2ban is now enabled by default in production mode

* Testing an option to disable remote syslogging

* Added SET_CSP_FRAME_SRC variable to extend the frame-src CSP setting

* Added SSL_FORCE_SELF_SIGNED env variable force generation of self signed certificates

* Added -H flag to pure-ftp to disable reverse DNS

* Added fail2ban jail for pureftpd

* Bug fix: added check to verify for malformed env variables passed via -e

* Bug fix: prevent fail2ban to lose the syslog sync in case of log rotation on the host

* Bug fix: reload swaggerui after ssl renewal

* Bump python-on-whales to 0.32.0

* Reimplemented logs command by using python on whales instead of compose v1

* Dropped compose v1 wrapper

* Dropped dependency with docker compose v1

* Renamed celery-beat service into celerybeat to prevent clashes with normal celery prefix (fix due 
after the upgrade to compose 2.0.0+ due to the separator change from _ to -)

* Bug fix to prevent remote syslogging in dev mode

* Status command can now be restricted to requested services

* Bug fix to prevent the check command to fail in case of uncommited files in custom builds

* Bug fix: run command with non existing services was not properly working

RAPyDo 2.0 (#36)

* Bump RAPyDo version to 2.0

* Dropped support for python 3.6, bumped min python version to 3.7

* Dropped support for docker 19, bumped min docker version to 20.10.0

* Replaced OrderedDict with dict

* Introduced python-on-whales and removed explicit dependency with docker.py

* Added swarm init and token check to init and check commands

* Dump command is no loger based on plumbum

* pre commit config upgrade

* Refactor of compose configuration data objects

* Compose configuration is now created on every command

* Builds module is now fully typed

* Implemented Swarm Mode with ability to automatically override commands

* Dropped unused start --no-detach flag

* Implemented status command in swarm mode

* Swarm init and token check are now only executed if swarm mode is enabled

* Implemented scale command in swarm mode

* Added stub of stop command in swarm mode (with an error message, since stop is not implemented in swarm)

* Dropped non standard logs --service option

* Dropped unused privileged mode

* Stub of shell command in swarm mode

* Stubs of logs and shell commands in swarm mode (can't be implemented due to limitations on python on whales)

* Added PROXIED_CONNECTION env variable to backend container

* Moved volumes configuration to an external yml to allow for easy overrides in case of nfs or other sharing setup (e.g. GlusterFS)

* Added DATA_DIR and MULTI_HOST_MODE variables

* Added nfs volumes configuration for multi hosts setup

* Nfs /exports renamed into /volumes

* Dropped unused celerybeat volume

* Added local docker registry when multi host is enabled in swarm mode

* Added automatic push of custom images on the local docker registry when multi host mode is enabled

* Added TLS certificates to registry container

* Added custom registry image to handle with TLS certificates generation

* Added deploy options (replicas, reservations cpus/memory) to each service

* Replaced explicit /data with DATA_DIR variable

* Dropped VANILLA_DIR env variable

* Renamed letsencrypt_certs volume into ssl_certs, dropped ssl_dhparam volume, dropped /etc/nginx/ssl mountpoint

* Added LOG_RETENTION env variable to control the log retention setting (180 days by default)

* Fixed mypy configuration for v0.9+

* Bug fix to prevent execution of swarm commands if the swarm init is still missing

* Bug fix to prevent swarm initializaation checks before rapydo init

* Added AUTH_VERIFY_PASSWORD_STRENGTH env variable to frontend container

* Removed option to disable the password strength checks, now it is will be ALWAYS enabled

* Moved swarm options to a dedicated configuration to prevent docker compose warnings when swarm mode is off

* Added API_AUTOSTART env variable to let the backend to automatically in testing mode

* Dropped --detach flag on shell command

* Reimplemented pull command with python on whales instead of compose

* Added env variable FORCE_SSR_SERVER_MODE to simulate server side rendering at browser level

* Commands are now partitioned between compose, swarm and general categories

* Dropped obsolete diagnostic command

* Reimplemented build command with buildkit

* Added join command in swarm mode

* Added checks on amount of cpu and memory configured for a deployment in swarm mode

* Renamed controller.gitter into controller.utilities.git

* Centralized handling of docker client in deploy.docker package

* Renamed controller.swarm into controller.deploy.swarm

* Renamed controller.compose into controller.deploy.compose

* Implemented the --remote option to connect docker to remote hosts

* Dropped unused dhparam command

* Added a service_verify utility to replace verify command during tests

* Upgraded GitPython from 3.1.14 to 3.1.18

* Extended check updates script to automatically fetch latest versions from docker hub

* Env variable VANILLA_PACKAGE renamed as PROJECT_NAME

* Added a check to verify the correct compose file version in compose configuration files

* Cleanup configuration.load_yaml_file

* Bug fix: default postgres shell command now specifies user and database

* Added healthchecks to all services

* Bug fix to handle status of health-checked containers

* Added HEALTHCHECK_INTERVAL variable (also used during test to reduce startup time)

* Backend health check is now only enabled in production mode

* Added FILE_LOGLEVEL env variable (defaulted to WARNING in normal mode and INFO in TESTING)

* Upgraded mariadb from 10.5.9 to 10.6.2

* Restored redundant services cleanup before building

* Added a check to verify available images before compose start

* Bug fix to prevent build of all services when a non-activated service is requested

* Bug fix to raise errors in case of pull of non activated services

* Bug fix ssl command to prevent erland errors when clearing the pem cache

* Bug fix: removed -proto_dist inet_tls options in dev mode

* Bug fix to include custom non extended images to build targets

* Extended check command to verify missing images

* Added validation checks before custom builds

* Upgraded docker-compose from 1.28.4 to 1.29.2

* Upgraded redis from 6.2.2 to 6.2.4

* Upgraded pushpin from 1.31.0 to 1.32.2

* Precommit update

* Added NEOMODEL_SLOW_QUERIES env to backend database

* Added check to verify the docker buildx availability

* Dropped dockerfile-parse dependency

* All functions are now type-hinted

* Added --disallow-untyped-calls flag to mypy workflow

* Enabled Flower 1.0.0

* Renamed CELERYUI* variables to FLOWER*

* Basic implementation of logs command in swarm mode

* Collapsed all networks into a single default network

* Added flower health check and fixed postgres and redis check to prevent expose of sensitive data

* Enabled build in swarm mode

* Replaced BaseExceptions with Exceptions

* Dropped start --force flag

* Dropped --net option from remove command in swarm mode

* Restricted compose dump to enabled services

* Restricted compose config dump to start and build commands

* Restart command always re-create the container to be consistent with swarm behaviour

* Added restart command to swarm mode

* Invalid services in -S/--skip-services are now refused

* Implemented remove of specific services in swarm mode

* Added option --wait to scale command in swarm mode

* Added colors to status command in swarm mode

* Added a warning if docker compose v2 is not found

* Custom images are now included to pull if a build context is not specified

* Implemented celery auto reload in dev mode

* Swarm mode: restart now scale back to 1 if the service was previously removed or scaled to 0

* Bug fix to prevent errors in swarm mode when the stack is not deployed

* Removed python packages version check at runtime (all are already listed in setup dependencies and all are pretty old and unlikely present on normal environments [compose 2017, requests 2015, pip 2018)

* Dropped unused check_python_package (and sub-functions)

* Bug fix flower health check in production mode was working improperly

* Compose V2 is now a mandatory dependency if swarm mode is enabled. A warning is raised if not found if normal mode.

* Reimplemented dump command on python on whales in swarm mode

* In swarm mode missing bind folders are now verified and automatically created

* Implemented install docker/compose/buildx commands

* Upgraded compose-cli from v2.0.0-beta.3 to v2.0.0-beta.6

* Added development and production compose yamls

* Refactor of compose files loading functions

* Added angular-development.yml compose config

* Dropped unused DOCKER_NETWORK_MODE env variable

* Added SSL_VERIFY_CLIENT env variable to enable nginx ssl client cert verify

* Fixed proxy healthcheck to be able to properly work in case of ssl client cert validation

* Switched SSL_VERIFY_CLIENT values from off/on to 0/1 to prevent issues with compose

* Forced installation of latest buildx version during tests

* Replaced a generic DockerException with NoSuchService

* Bug fix: valid project names requires at least two characters

* Upgraded python-on-whales to version 0.23.0

* Implemented logs --follow in swarm mode

* Bug fix to restore consumed $ in healthchecks when dumping the configuration

* Added --state_save_interval=10000 option to flower

* Added missing DOMAIN env variable to flower container

* Added py.typed marker

* Set resolve_image=never on stack.deploy

* Replaced Application.exit with print_and_exit

* Added multihost test

* Introduced docker-machine on GA to test multi host deployments. The coniguration is currently disabled due to the lack of nested virtualization in GA instances (VT-x is not available)

* Added check to prevent scale to 2+ instances of non guaranteed services

* Added advertise_address option to swarm init (defaulted to local IP)

* Renamed ACTIVATE_SQLALCHEMYUI into ACTIVATE_ADMINER

* Increased containers start period from 30s to 5m

* Added container name to status output

* Enabled registry authentication

* Dropped MULTI_HOST_MODE variable

* Registry service is now configured outside the stack to be shared among projects

* Implemented rapydo registry command to run execute the registry service (temporary command, to be merged with interfaces and volatile in a near future)

* Added rapydo images command to query the local registry

* Fixed pull, build and start commands in swarm mode to use the local registry

* Added insecure-registries to daemon.json on GA

* Moved registry certificates from ssl_certs to registry volume (prevent nfs errors in multi host mode)

* Enabled Faker type hints

* Created two separated network for compose and swarm services to prevent collisions

* Untrusted registry TLS certificate now raises an error before starting pull and build

* Enabled registry healthcheck

* Added random REGISTRY_HTTP_SECRET to the registry container

* Upgraded buildx from 0.5.1 to 0.6.1

* Added checksum verification of downloaded installation script/bin for docker,compose and buildx

* Compatibility fixes on windows

* Bug fix to allow SWARM_MANAGER_ADDRESS as -e option

* Added registry and images tests

* Added MAX_REQUESTS_PER_SECOND_AUTH MAX_REQUESTS_PER_SECOND_API and MAX_REQUESTS_PER_SECOND_ASSETS variables

RAPyDo 1.2 (#35)

* Bump RAPyDo version to 1.2

* Enabled support for Docker BuildKit on build command

* Added .prettierignore as mandatory file

* Set backend models emails folder as mandatory

* Dropped deprecated --command options from shell and volatile commands (command is now passed as a positional argument)

* Upgraded Redis from 6.0.10 to 6.2.2

* Upgraded GitPython from 3.1.13 to 3.1.14

* Pre-commit autoupdates

* Replaced css with scss files

* Extended create command to support backend with no authentication service

* Added AUTH_ENABLE varable to frontend container

* Added ssl path to backend and celery containers

* Centralized pre-commit configuration for all projects, deprecated local configurations

* Dropped ancestors command (no longer compatible with BuildKit)

* Dropped ssl --force flag

* Dropped LETSENCRYPT_MODE env variable

* Added option --skip-services/-S option

* Dropped rapydo installation ad user level when no-editable mode is enable to prevent pip misconfiguration between global and local environments

* Added more informative output to rapydo install to explain the actions and commands that will be executed and to give the user the opportunity to install the controller by himself

* Replaced mongoui interface with adminer. Deprecated swagger, celery and sqlalchemy interfaces, replaced respectively with swaggerui, flower and adminer

* Added data imports volume to backend and neo4j

* Deprecated support to python 3.6

* Added default command for mariadb containers

* Implemented mariadb backup and restore

* Extended projectrc creation with env variables passed with -e option

* Implemented RabbitMQ backend and restore

* Implemented Redis backup and restore

* Bumped min required docker version to 18.09.2 (to ensure the fix for CVE-2019-5736 vulnerability). Bumped the recommended docker version from 19.03.8 to 19.03.14

* Added SET_MAX_REQUESTS_PER_SECOND_API/AUTH/ASSETS variables

* Added github actions templates, also included in add command

RAPyDo 1.1 (#34)

* Bump RAPyDo version to 1.1

* Upgraded isort from 5.5.2 to 5.7.0

* Upgraded prettier from 2.1.1 to 2.2.1

* Upgraded pyupgrade from v2.7.2 to v2.10.0

* Upgraded flake8 from 3.8.3 to 3.8.4

* Upgraded mypy from v0.790 to v0.812

* Upgraded redis from 6.0.9 to 6.0.10

* Upgraded GitPython from 3.1.11 to 3.1.13

* Upgraded PyYAML from 5.3.1 to 5.4.1

* Invalidated all project names containing special characters, number or upper case characters (i.e. only lower case characters are now allowed, as specified in PEP8 for Package and Module Names

* Upgraded Compose from 1.27.4 to 1.28.4

* Automatically set new compose --no-log-prefix flag to logs command (set False when a single service is displayed)

* Fixed rapydo dump command after upgrade to compose 1.28+

* Added initialize_testing_environment to initializer class

* Enabled Group Coordinator Role

* Numbers are now allowed in project name if not leading

* Upgraded mariadb from 10.5.8 to 10.5.9

* Added upload path to celery container

* Required a __init__.py into the main backend folder to let mypy to correctly extract typing from the project module

* Bump min recommended docker version from 19.03.1 to 19.03.8

* Mapped data/uploads path on /uploads in backend and celery container by default

* Added data/uploads to folders created at init time

* Switched deprecation warning from normal logging to warnings module

* Added tests on invalid characters in mongodb passwords

* jwt_tokens volume renamed to secrets

* Added tests on Packages module

* Added checks on controller installation path to inform user if the update command will also have updated the controller or not

* Added a preliminary check on updating repositories to prevent partial updating

* pre-commit autoupdate

* Added AUTH_TOTP_VALIDITY_WINDOW env variable (defaulted to int value 1)

* Increased default login ban time from 1 hour to 12 to leverage the implementation of the unlock tokens

* Added a simple SMTP server for testing purpose (or to be used when a real smtp server can't be used)

* Added option to start neo4j in recovery mode

* Added project keywords

* Added forbid-html-img-without-alt-text to pre-commit config template

* Added options --dry-run and --max to backup command

* CELERY_ENABLE_CONNECTOR is now set at 0 by default to have a more fine grained controller over the celery activation

* Removed CELERYBEAT_ENABLE_CONNECTOR variable

* Added SHOW_LOGIN env variable

* Enabled docker BuildKit during tests

RAPyDo 1.0 (#33)

* Bump RAPyDo version to 1.0

* Added a sleep when using shell and volatile commands with deprecated --command option to make more visible the deprecation warning

* Secured Redis with a password

* Upgraded mariadb from 10.5.6 to 10.5.8

* Upgraded mongo from 4.4.1 to 4.4.2

* Upgraded dockerfile-parse from 1.0.0 to 1.1.0

* Upgraded pushpin from 1.30.0 to 1.31.0

* Added python 3.9 tests

* Added __init__.py to required files for each required folder in backend

* Moved browserslist to package.json

* Deprecated browserslist files in data frontend dir

* Deprecated boolean env variabiles (True, true, False and false) and back to old-fashioned 0/1 values. Backend and Frontend use different booleans due to Py vs Js 0/1 is a much more portable value to prevent true|True|true This fixes troubles in setting boolean values only used by Angular (expected true|false) or used by Pyton (expected True|False)

* Added simple test workflow based on dump + compose

* Replaced CORS_ALLOW_ALL_ORIGINS variable (expected 0/1) with CORS_ALLOWED_ORIGIN (expected empty, * or any hostname)

* Added ENABLE_ANGULAR_SSR variable and tsconfig.server.json file mapping to angular container

* Default ENABLE_FOOTER value changed from 0 to 1

* Removed unused AUTH_REGISTER_FAILED_LOGIN env variable

* Added AUTH_LOGIN_BAN_TIME to backend container (default to 3600 seconds)

* Added warning for short password (production mode only)

* Added test for short passwords warning

* Set default user and password for mongoDB

* Added variable mappings for mongo user and password

* MongoDB container is now based on custom mongodb image to properly handle user management

* Added swaggerui authentication

* Added checks on invalid characters in REDIS password

* Added SET_CSP_CONNECT_SRC env variable

* Added ALEMBIC_AUTO_MIGRATE env variable to enable/disable automatic flask migrations

* Added tsconfig.server.json to required frontend files

* Removed query url option from suggested swaggerui url (now automatically embedded at runtime)

* SSR flag is now enabled by default

* AUTH_SECOND_FACTOR_AUTHENTICATION Changed from TOTP string to boolean

* Added AUTH_TOKEN_IP_GRACE_PERIOD env variable

* Added AUTH_MAX_LOGIN_ATTEMPTS and AUTH_LOGIN_BAN_TIME to cypress

* Enabled test on redis invalid characters

* Reduced default token grace period from 7200 to 1800 seconds

* Removed fake fixture to switch to standard faker fixture

RAPyDo 0.9 (#32)

* Bump RAPyDo version to 0.9

* Added .gitattributes to expected files

* Split Initializer and Customizer classes into dedicated modules

* Renamed custom.project.options.ts into customization.ts

* Fix celery cli options to allow the upgrade to Celery 5+

* Dropped unused Packages.check_version utility

* Added templated favicons

* Added project raw_files to expected files verified at startup

* Upgraded redis from 6.0.7 to 6.0.9

* Upgraded mariadb from 10.5.5 to 10.5.6

* Upgraded swagger-ui from v3.32.5 to v3.36.0

* Upgraded docker-compose from 1.27.3 to 1.27.4

* Upgraded GitPython from 3.1.8 to 3.1.11

* Marked assets/favicon.ico as obsolete

* group_coordinator role is now disabled by default

* Adding static type checking

* Extended rapydo check to raise warnings when files intended to be immutable (like .gitattributes, .pre-commit-config.yaml and pyproject.toml) differ from the corresponding template

* Bug fix to prevent 'rapydo upgrade --path' to overwrite raw files

* Added variables for default expiration and verification time used by connectors

* Added support for crontab enabling in backend container

* Cron folder moved from data/cron to projects/$/backend/cron

* Added variables GZIP_COMPRESSION_ENABLE, GZIP_COMPRESSION_THRESHOLD, GZIP_COMPRESSION_LEVEL to configure gzip compression of backend responses

* Added RABBITMQ_MANAGEMENT_PORT to backend container to allow rabbit connector to communicate with rabbit management APIs

* Added DOMAIN_ALIASES variable to proxy container

* Dropped support for Ubuntu 14.04 (already reached EOL in April 2019)

* Utilities modules (configuration, services, system) are now fully typed

* SQLAlchemyUI (adminer) is now based on custom rapydo/adminer image to include SSL support in production mode

* Fix to ensure image pull before launching interfaces. Remove suppress_stdout before launching interfaces

* Added custom SwaggerUI build to include SSL support in production mode

* Added DOMAIN variable to swaggerui container

* Moved tests from travis to GitHub Actions. Dropped travis configuration

* Added --no-tty option to rapydo verify command

* Added --force flag to rapydo add command

* Added on a custom action to install controller during tests (install action implemented in a dedicated repository)

* Extended coverage to tests files

* Added option --add-tests to rapydo add

* Implemented rapydo add endpoint [name] --add-tests to include a test scaffold to created endpoint

* Set timeout-minutes to github actions configuration

* Added workflow_dispatch to GA workflow

* Implemented rapydo add component [name] --add-tests to include a test scaffold to created endpoint

* Implemented rapydo add integration_test [name]

* Included route when creating sink component (special case used by frontend tests)

* Extended special sink component with custom template used during angular tests

* Removed obsolete messages level

* Added AUTH_MIN_PASSWORD_LENGTH to angular container

* Set min docker recommended version to 19.03.1

* Added Redis connector

* Fixed containers restart policies and added tests on SSL mandatory files

* Added automatic clearing of the rabbitmq pem cache after issuing a new certificate

* Added Faker to test suite

* Added --quiet flag to pull command