Move tools/ci_* projects to airbyte-ci, update to use Poetry, bump to python 3.10

airbyte-ci/ci/ci_common_utils/pyproject.toml

@@ -0,0 +1,25 @@
+#
+# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
+#
+
+[tool.poetry]
+name = "ci_common_utils"
+version = "0.0.0"
+description = "Suite of all often used classes and common functions"
+authors = ["Airbyte <contact@airbyte.io>"]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+cryptography = "^3.4.7"
+requests = "^2.28.2"
+pyjwt = "^2.1.0"
+
+[tool.poetry.group.test.dependencies]
+pytest = "^7.2.2"
+
+[tool.poetry.group.dev.dependencies]
+requests-mock = "^1.9.3"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"

tools/ci_connector_ops/README.md → airbyte-ci/ci/ci_connector_ops/README.md

@@ -15,7 +15,7 @@ development environment of choice. To activate it from the terminal, run:
 
 ```bash
 source .venv/bin/activate
-pip install -e . # assuming you are in the ./tools/ci_connector_ops directory
+pip install -e . # assuming you are in the ./airbyte-ci/ci_connector_ops directory
 ```
 
 pip will make binaries for all the commands in setup.py, so you can run `allowed-hosts-checks` directly from the virtual-env
@@ -31,5 +31,5 @@ pip install -e ".[tests]"
 Unit tests are currently configured to be run from the base `airbyte` directory. You can run the tests from that directory with the following command:
 
 ```
-pytest -s tools/ci_connector_ops/tests
+pytest -s airbyte-ci/ci_connector_ops/tests
 ```

airbyte-ci/ci/ci_connector_ops/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry]
+name = "ci_connector_ops"
+version = "0.2.1"
+description = "Packaged maintained by the connector operations team to perform CI for connectors"
+authors = ["Airbyte <contact@airbyte.io>"]
+
+[tool.poetry.dependencies]
+python = "^3.10"
+click = "^8.1.3"
+requests = "^2.28.2"
+PyYAML = "^6.0"
+GitPython = "^3.1.29"
+pydantic = "^1.9"
+PyGithub = "^1.58.0"
+rich = "^11.0.1"
+pydash = "^7.0.4"
+google-cloud-storage = "^2.8.0"
+ci-credentials = {path = "../ci_credentials"}
+
+[tool.poetry.group.dev.dependencies]
+pyinstrument = "^0.24.4"
+
+[tool.poetry.group.test.dependencies]
+pytest = "^6.2.5"
+pytest-mock = "^3.10.0"
+freezegun = "^1.1.0"
+
+[tool.poetry.scripts]
+check-test-strictness-level = "ci_connector_ops.acceptance_test_config_checks:check_test_strictness_level"
+write-review-requirements-file = "ci_connector_ops.acceptance_test_config_checks:write_review_requirements_file"
+print-mandatory-reviewers = "ci_connector_ops.acceptance_test_config_checks:print_mandatory_reviewers"
+allowed-hosts-checks = "ci_connector_ops.allowed_hosts_checks:check_allowed_hosts"
+run-qa-checks = "ci_connector_ops.qa_checks:run_qa_checks"

tools/ci_credentials/README.md → airbyte-ci/ci/ci_credentials/README.md

@@ -3,18 +3,30 @@ CLI tooling to read and manage GSM secrets:
 - `write-to-storage` download a connector's secrets locally in the connector's `secret` folder
 - `update-secrets` uploads new connector secret version that were locally updated.
 
+## Requirements
 
-## Development
+This project requires Python 3.10 and pipx.
+
+## Installation
 
-Set up the world the same way Google Actions does it in `test-command.yml`.
+The recommended way to install `ci_credentials` is using pipx. This ensures the tool and its dependencies are isolated from your other Python projects.
 
+If you haven't installed pipx, you can do it with pip:
+
+```bash
+python -m pip install --user pipx
+python -m pipx ensurepath
 ```
-source venv/bin/activate
-pip install --quiet tox==3.24.4
-tox -r -c ./tools/tox_ci.ini
-pip install --quiet -e ./tools/ci_*
+
+Once pipx is installed, navigate to the root directory of the project, then run:
+
+```bash
+pipx install airbyte-ci/ci/ci_credentials/
 ```
 
+This command installs ci_credentials and makes it globally available in your terminal.
+
+## Get GSM access
 Download a Service account json key that has access to Google Secrets Manager.
 
 ### Create Service Account
@@ -30,35 +42,44 @@ Download a Service account json key that has access to Google Secrets Manager.
 
 ### Setup ci_credentials
 * In your .zshrc, add: `export GCP_GSM_CREDENTIALS=$(cat <path to JSON file>)`
-* Follow README.md under `tools/ci_credentials`
 
-After making a change, you have to reinstall it to run the bash command: `pip install --quiet -e ./tools/ci_*`
+## Development
+During development, you can use the `--editable` option to make changes to the `ci_credentials` package and have them immediately take effect without needing to reinstall the package:
+
+```bash
+pipx install --editable airbyte-ci/ci/ci_credentials/
+```
+
+This is useful when you are making changes to the package and want to test them in real-time.
+
+## Usage
+After installation, you can use the ci_credentials command in your terminal.
 
 ## Run it
 
 The `VERSION=dev` will make it so it knows to use your local current working directory and not the Github Action one.
 
 ### Help
 ```bash
-ci_credentials --help
+VERSION=dev ci_credentials --help
 ```
 
 ### Write credentials for a specific connector to local storage
 To download GSM secrets to `airbyte-integrations/connectors/source-bings-ads/secrets`:
 ```bash
-ci_credentials source-bing-ads write-to-storage
+VERSION=dev ci_credentials source-bing-ads write-to-storage
 ```
 
 ### Write credentials for all connectors to local storage
 To download GSM secrets to for all available connectors into their respective `secrets` directories:
 ```bash
-ci_credentials all write-to-storage
+VERSION=dev ci_credentials all write-to-storage
 ```
 
 ### Update secrets
 To upload to GSM newly updated configurations from `airbyte-integrations/connectors/source-bings-ads/secrets/updated_configurations`:
 
 ```bash
-ci_credentials source-bing-ads update-secrets
+VERSION=dev ci_credentials source-bing-ads update-secrets
 ```