Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding terraform and tests #371

Merged
merged 36 commits into from
Jul 12, 2023
Merged
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
d8bc049
adding terraform and tests
franklinWhaite Jul 6, 2023
2e14dd1
Merge branch 'master' into master
franklinWhaite Jul 11, 2023
f98521d
Update tools/cloud_functions/bq_table_snapshots/terraform/backend.tf
franklinWhaite Jul 11, 2023
8d9234e
Update tools/cloud_functions/bq_table_snapshots/terraform/main.tf
franklinWhaite Jul 11, 2023
c7a5b54
Update tools/cloud_functions/bq_table_snapshots/terraform/main.tf
franklinWhaite Jul 11, 2023
e8ec829
Update tools/cloud_functions/bq_table_snapshots/terraform/backend.tf
franklinWhaite Jul 11, 2023
f88db34
Update tools/cloud_functions/bq_table_snapshots/tests/bq_backup_fetch…
franklinWhaite Jul 11, 2023
c0a930d
Update tools/cloud_functions/bq_table_snapshots/tests/bq_backup_creat…
franklinWhaite Jul 11, 2023
4eb7585
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 11, 2023
8501dfc
Update tools/cloud_functions/bq_table_snapshots/tests/bq_backup_creat…
franklinWhaite Jul 11, 2023
fd48691
Update tools/cloud_functions/bq_table_snapshots/terraform/function.tf
franklinWhaite Jul 11, 2023
856c8a3
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 11, 2023
b172b36
Update tools/cloud_functions/bq_table_snapshots/tests/bq_backup_creat…
franklinWhaite Jul 11, 2023
edfd098
Update tools/cloud_functions/bq_table_snapshots/tests/bq_backup_fetch…
franklinWhaite Jul 11, 2023
6525b7a
using .tfvars and adding gcs backend
franklinWhaite Jul 12, 2023
7a5018e
updating readme, using .tfvars, adding gcs backend
franklinWhaite Jul 12, 2023
cf01080
adding descriptions for variables
franklinWhaite Jul 12, 2023
f38219d
removing tf state files
franklinWhaite Jul 12, 2023
b178a40
Update tools/cloud_functions/bq_table_snapshots/terraform/deployment_…
franklinWhaite Jul 12, 2023
3f8f8bc
Update tools/cloud_functions/bq_table_snapshots/terraform/deployment_…
franklinWhaite Jul 12, 2023
ec95133
Update tools/cloud_functions/bq_table_snapshots/README.md
franklinWhaite Jul 12, 2023
18c178c
Update tools/cloud_functions/bq_table_snapshots/README.md
franklinWhaite Jul 12, 2023
67b2671
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
086428a
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
7d45fda
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
fa9bd01
fixing typos
franklinWhaite Jul 12, 2023
f3c5e20
renaiming .tfvars file
franklinWhaite Jul 12, 2023
7308c3c
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
042d060
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
4be091d
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
9c665ae
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
afe5e2e
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
232f7c0
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
2bf70ea
reafactoring readme
franklinWhaite Jul 12, 2023
067b4dc
Update tools/cloud_functions/bq_table_snapshots/terraform/variables.tf
franklinWhaite Jul 12, 2023
243e9d4
Update tools/cloud_functions/bq_table_snapshots/terraform/backend.tf
franklinWhaite Jul 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reafactoring readme
  • Loading branch information
franklinWhaite committed Jul 12, 2023
commit 2bf70eab191d785282590c163e701dc7bbe43d03
9 changes: 0 additions & 9 deletions tools/cloud_functions/bq_table_snapshots/README.md
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 16 replace Scheduloer with Scheduler

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 35 has typo: ame
Fix: name

Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,8 @@ The **bq-snap-start-process** Cloud Scheduler Job will run monthly and trigger t
## bq_backup_fetch_tables_names
The **bq_backup_fetch_tables_names** cloud function will fetch all the table names in source_dataset_name. It will then apply filters based on tables_to_include_list and tables_to_exclude_list to determine the tables in scope. Finally, it will submit one Pub/Sub message per table.

The Cloud Function uses the following environment variables to determine which BigQuery project to use for compute and which to use for storing your BigQuery snapshots:
* `DATA_PROJECT_ID` id of project used for BQ storage
* `PUBSUB_PROJECT_ID` id of project with P/S topic
* `TABLE_NAME_PUBSUB_TOPIC_ID` name of P/S topic where this code will publish to

## bq_backup_create_snapshots
The **bq_backup_create_snapshots** Cloud Function will submit a BigQuery job to create a snapshot for each table in scope. This Cloud Function will suffix the snapshot name with the snapshot datetime to guarantee a unique name. It will also calculate and set the expiration time of the snapshot based on seconds_before_expiration. Finally, it will determine the snapshot time based on crontab_format.
The following environment variables must be set:
* `BQ_DATA_PROJECT_ID` id of project used for BQ storage
* `BQ_JOBS_PROJECT_ID` id of project used for BQ compute


### About the crontab_format field
If DATASET_1 has 500 tables, 500 Pub/Sub messages are sent, and 500 Cloud Function invocations are performed. If the Cloud Function used the current time when it creates the snapshots then these 500 snapshots will represent different points in time. To avoid this the Cloud Function will create the snapshots for the table as they were when the Cloud Scheduler job (bq-snap-start-process) was triggered. To achieve this the Cloud Function will calculate the previous interval based on **crontab_format**.
Expand Down