Instructions for contributing (dbt-labs#104)

* Instructions for contributing

* Keep consistency between spaces and tabs

* Update changelog

.gitignore

@@ -3,3 +3,5 @@ target/
 dbt_modules/
 dbt_packages/
 logs/
+env/
+integration_tests/profiles.yml

CHANGELOG.md

@@ -17,6 +17,7 @@ This macro generates a series of terminal commands (appended w) bash script whic
 
 ## Quality of life
 - Addition of the [base_model_creation](bash_scripts/base_model_creation.sh) bash script which allows users to input multiple tables as a list and generate a terminal command that will combine **all** [create_base_models](macros/create_base_models.sql) commands. This way, you can generate base models for all your sources at once.
+- Instructions for contributing ([#99](https://github.com/dbt-labs/dbt-codegen/issues/99), [#104](https://github.com/dbt-labs/dbt-codegen/pull/104))
 
 ## Contributors:
 - [@fivetran-joemarkiewicz](https://github.com/fivetran-joemarkiewicz) (#83)

CONTRIBUTING.md

@@ -0,0 +1,80 @@
+# Contributing to `dbt-codegen`
+
+`dbt-codegen` is open source software. It is what it is today because community members have opened issues, provided feedback, and [contributed to the knowledge loop](https://www.getdbt.com/dbt-labs/values/). Whether you are a seasoned open source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.
+
+Remember: all PRs (apart from cosmetic fixes like typos) should be [associated with an issue](https://docs.getdbt.com/docs/contributing/oss-expectations#pull-requests).
+
+1. [About this document](#about-this-document)
+1. [Getting the code](#getting-the-code)
+1. [Setting up an environment](#setting-up-an-environment)
+1. [Implementation guidelines](#implementation-guidelines)
+1. [Testing dbt-codegen](#testing)
+1. [Adding CHANGELOG Entry](#adding-changelog-entry)
+1. [Submitting a Pull Request](#submitting-a-pull-request)
+
+## About this document
+
+There are many ways to contribute to the ongoing development of `dbt-codegen`, such as by participating in discussions and issues. We encourage you to first read our higher-level document: ["Expectations for Open Source Contributors"](https://docs.getdbt.com/docs/contributing/oss-expectations).
+
+The rest of this document serves as a more granular guide for contributing code changes to `dbt-codegen` (this repository). It is not intended as a guide for using `dbt-codegen`, and some pieces assume a level of familiarity with Python development (virtualenvs, `pip`, etc). Specific code snippets in this guide assume you are using macOS or Linux and are comfortable with the command line.
+
+### Notes
+
+- **CLA:** Please note that anyone contributing code to `dbt-codegen` must sign the [Contributor License Agreement](https://docs.getdbt.com/docs/contributor-license-agreements). If you are unable to sign the CLA, the `dbt-codegen` maintainers will unfortunately be unable to merge any of your Pull Requests. We welcome you to participate in discussions, open issues, and comment on existing ones.
+- **Branches:** All pull requests from community contributors should target the `main` branch (default). If the change is needed as a patch for a version of `dbt-codegen` that has already been released (or is already a release candidate), a maintainer will backport the changes in your PR to the relevant branch.
+
+## Getting the code
+
+### Installing git
+
+You will need `git` in order to download and modify the `dbt-codegen` source code. On macOS, the best way to download git is to just install [Xcode](https://developer.apple.com/support/xcode/).
+
+### External contributors
+
+If you are not a member of the `dbt-labs` GitHub organization, you can contribute to `dbt-codegen` by forking the `dbt-codegen` repository. For a detailed overview on forking, check out the [GitHub docs on forking](https://help.github.com/en/articles/fork-a-repo). In short, you will need to:
+
+1. Fork the `dbt-codegen` repository
+2. Clone your fork locally
+3. Check out a new branch for your proposed changes
+4. Push changes to your fork
+5. Open a pull request against `dbt-labs/dbt-codegen` from your forked repository
+
+### dbt Labs contributors
+
+If you are a member of the `dbt-labs` GitHub organization, you will have push access to the `dbt-codegen` repo. Rather than forking `dbt-codegen` to make your changes, just clone the repository, check out a new branch, and push directly to that branch.
+
+## Setting up an environment
+
+There are some tools that will be helpful to you in developing locally. While this is the list relevant for `dbt-codegen` development, many of these tools are used commonly across open-source python projects.
+
+### Tools
+
+These are the tools used in `dbt-codegen` development and testing:
+- [`make`](https://users.cs.duke.edu/~ola/courses/programming/Makefiles/Makefiles.html) to run multiple setup or test steps in combination. Don't worry too much, nobody _really_ understands how `make` works, and our Makefile aims to be super simple.
+- [CircleCI](https://circleci.com/) for automating tests and checks, once a PR is pushed to the `dbt-codegen` repository
+
+A deep understanding of these tools in not required to effectively contribute to `dbt-codegen`, but we recommend checking out the attached documentation if you're interested in learning more about each one.
+
+## Testing
+
+Once you're able to manually test that your code change is working as expected, it's important to run existing automated tests, as well as adding some new ones. These tests will ensure that:
+- Your code changes do not unexpectedly break other established functionality
+- Your code changes can handle all known edge cases
+- The functionality you're adding will _keep_ working in the future
+
+See here for details for running existing integration tests and adding new ones:
+- [integration_tests/README.md](integration_tests/README.md)
+
+## Adding CHANGELOG Entry
+
+Unlike `dbt-core`, we edit the `CHANGELOG.md` directly.
+
+You don't need to worry about which `dbt-codegen` version your change will go into. Just create the changelog entry at the top of CHANGELOG.md and open your PR against the `main` branch. All merged changes will be included in the next minor version of `dbt-codegen`. The maintainers _may_ choose to "backport" specific changes in order to patch older minor versions. In that case, a maintainer will take care of that backport after merging your PR, before releasing the new version of `dbt-codegen`.
+
+## Submitting a Pull Request
+
+A `dbt-codegen` maintainer will review your PR. They may suggest code revision for style or clarity, or request that you add unit or integration test(s). These are good things! We believe that, with a little bit of help, anyone can contribute high-quality code.
+
+Automated tests run via CircleCI. If you're a first-time contributor, all tests (including code checks and unit tests) will require a maintainer to approve. Changes in the `dbt-codegen` repository trigger integration tests.
+
+Once all tests are passing and your PR has been approved, a `dbt-codegen` maintainer will merge your changes into the active development branch. And that's it! Happy developing :tada:

Makefile

@@ -0,0 +1,24 @@
+.DEFAULT_GOAL:=help
+
+.PHONY: test
+test: ## Run the integration tests.
+	@./run_test.sh $(target)
+
+.PHONY: dev
+dev: ## Installs dbt-* packages in develop mode along with development dependencies.
+	@\
+	echo "Install dbt-$(target)..."; \
+	python -m pip install --upgrade pip setuptools; \
+	python -m pip install --pre dbt-core "dbt-$(target)";
+
+.PHONY: setup-db
+setup-db: ## Setup Postgres database with docker-compose for system testing.
+	@\
+	docker-compose up --detach postgres
+
+.PHONY: help
+help: ## Show this help message.
+	@echo 'usage: make [target]'
+	@echo
+	@echo 'targets:'
+	@grep -E '^[8+a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

docker-compose.yml

@@ -0,0 +1,8 @@
+version: "3.7"
+services:
+  postgres:
+    image: cimg/postgres:9.6
+    environment:
+      - POSTGRES_USER=root
+    ports:
+      - "5432:5432"

integration_tests/.env/bigquery.env

@@ -0,0 +1,2 @@
+BIGQUERY_SERVICE_KEY_PATH=
+BIGQUERY_TEST_DATABASE=

integration_tests/.env/postgres.env

@@ -0,0 +1,5 @@
+POSTGRES_TEST_HOST=localhost
+POSTGRES_TEST_USER=root
+POSTGRES_TEST_PASS=''
+POSTGRES_TEST_PORT=5432
+POSTGRES_TEST_DBNAME=circle_test

integration_tests/.env/redshift.env

@@ -0,0 +1,5 @@
+REDSHIFT_TEST_HOST=
+REDSHIFT_TEST_USER=
+REDSHIFT_TEST_PASS=
+REDSHIFT_TEST_DBNAME=
+REDSHIFT_TEST_PORT=

integration_tests/.env/snowflake.env

@@ -0,0 +1,6 @@
+SNOWFLAKE_TEST_ACCOUNT=
+SNOWFLAKE_TEST_USER=
+SNOWFLAKE_TEST_PASSWORD=
+SNOWFLAKE_TEST_ROLE=
+SNOWFLAKE_TEST_DATABASE=
+SNOWFLAKE_TEST_WAREHOUSE=

integration_tests/README.md

@@ -0,0 +1,141 @@
+### Overview
+1. Prerequisites
+1. Configure credentials
+1. Setup Postgres (optional)
+1. Setup virtual environment
+1. Installation for development
+1. Run the integration tests
+1. Run tests
+1. Creating a new integration test
+
+### Prerequisites
+- python3
+- Docker (optional)
+
+### Configure credentials
+Edit the env file for your TARGET in `integration_tests/.env/[TARGET].env`.
+
+Load the environment variables:
+```shell
+set -a; source integration_tests/.env/[TARGET].env; set +a
+```
+
+or more specific:
+```shell
+set -a; source integration_tests/.env/postgres.env; set +a
+```
+
+#### Setup Postgres (optional)
+
+Docker and `docker-compose` are both used in testing. Specific instructions for your OS can be found [here](https://docs.docker.com/get-docker/).
+
+Postgres offers the easiest way to test most `dbt-codegen` functionality today. Its tests are the fastest to run, and the easiest to set up. To run the Postgres integration tests, you'll have to do one extra step of setting up the test database:
+
+```shell
+make setup-db
+```
+or, alternatively:
+```shell
+docker-compose up --detach postgres
+```
+
+### Setup virtual environment
+
+We strongly recommend using virtual environments when developing code in `dbt-codegen`. We recommend creating this virtualenv
+in the root of the `dbt-codegen` repository. To create a new virtualenv, run:
+```shell
+python3 -m venv env
+source env/bin/activate
+```
+
+This will create and activate a new Python virtual environment.
+
+### Installation for development
+
+First make sure that you set up your virtual environment as described above. Also ensure you have the latest version of pip and setuptools installed:
+```
+python -m pip install --upgrade pip setuptools
+```
+
+Next, install `dbt-core` (and its dependencies) with:
+
+```shell
+make dev target=[postgres|redshift|...]
+# or
+pip install --pre dbt-core dbt-[postgres|redshift|...]
+```
+
+or more specific:
+
+```shell
+make dev target=postgres
+# or
+pip install --pre dbt-core dbt-postgres
+```
+
+### Run the integration tests
+
+To run all the integration tests on your local machine like they will get run in the CI (using CircleCI):
+
+```shell
+make test target=[postgres|redshift|...]
+# or
+./run_test.sh [postgres|redshift|...]
+```
+
+or more specific:
+
+```shell
+make test target=postgres
+# or
+./run_test.sh postgres
+```
+
+Where possible, targets can run in docker containers (this works for Postgres or in the future Spark for example). For managed services like Snowflake, BigQuery and Redshift this is not possible, hence your own configuration for these services has to be provided in the appropriate env files in `integration_tests/.env/[TARGET].env`
+
+### Creating a new integration test
+
+#### Set up profiles
+Do one of the following:
+1. Use the `profiles.yml` in the current working directory for dbt Core 1.3 and above
+    ```shell
+    cp integration_tests/ci/sample.profiles.yml integration_tests/profiles.yml
+    ```
+1. Use `DBT_PROFILES_DIR`
+    ```shell
+    cp integration_tests/ci/sample.profiles.yml integration_tests/profiles.yml
+    export DBT_PROFILES_DIR=$(cd integration_tests && pwd)
+    ```
+1. Use `~/.dbt/profiles.yml`
+    - Copy contents from `integration_tests/ci/sample.profiles.yml` into `~/.dbt/profiles.yml`.
+
+#### Add your integration test
+This directory contains an example dbt project which tests the macros in the `dbt-codegen` package. An integration test typically involves making:
+1. a new seed file
+2. a new model file
+3. a generic test to assert anticipated behaviour.
+
+For an example of integration tests, check out the tests for the `get_url_parameter` macro in the `dbt-utils` project:
+
+1. [Macro definition](https://github.com/dbt-labs/dbt-utils/blob/main/macros/web/get_url_parameter.sql)
+2. [Seed file with fake data](https://github.com/dbt-labs/dbt-utils/blob/main/integration_tests/data/web/data_urls.csv)
+3. [Model to test the macro](https://github.com/dbt-labs/dbt-utils/blob/main/integration_tests/models/web/test_urls.sql)
+4. [A generic test to assert the macro works as expected](https://github.com/dbt-labs/dbt-utils/blob/main/integration_tests/models/web/schema.yml)
+
+Once you've added all of these files, you should be able to run:
+
+Assuming you are in the `integration_tests` folder,
+```shell
+dbt deps --target {your_target}
+dbt seed --target {your_target}
+dbt run --target {your_target} --model {your_model_name}
+dbt test --target {your_target} --model {your_model_name}
+```
+
+Alternatively:
+```shell
+dbt deps --target {your_target}
+dbt build --target {your_target} --select +{your_model_name}
+```
+
+If the tests all pass, then you're good to go! All tests will be run automatically when you create a PR against this repo.

run_test.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+echo `pwd`
+cd integration_tests
+cp ci/sample.profiles.yml profiles.yml
+
+dbt --warn-error deps --target $1 || exit 1
+dbt --warn-error run-operation create_source_table --target $1 || exit 1
+dbt --warn-error seed --target $1 --full-refresh || exit 1
+dbt --warn-error run --target $1 || exit 1
+dbt --warn-error test --target $1 || exit 1