Adding BigQuery optimization scripts (GoogleCloudPlatform#376)

* add optimization scripts

* Add script to README for enabling recommender API and add new script for investigating tables with high freq DML

* add comments

* Adding comments to existing scripts

* add default to variable

* linting

* add script for harvesting query perf insights

* nit

* improve and add comments

* remove query hashing UDF and instead use existing query hash field in the info schema views

* Update docs

* remove unnecessary check

* update largest tables scripts (GoogleCloudPlatform#375)

* Add frequent_daily_table_dml to README and nits

* markdown formatting

* revert formatting

* remove experimental script

* standardizing scripts to use the same num_days_to_scan variable

* refactor scripts to not do any sorting

* add missing INSERT

* revert

* remove unnecessary INSERTs and use bqutil UDF

* comments fixes

* fix regex on nonoptimal view script

* run table_read_patterns script before other scripts

* remove unnecessary alias

* remove unnecessary filter

* only check for completed queries

* fix perf insights sql

* simplify queries_grouped_by_hash.sql by making it query over the jobs_by_org view and removed a lot of the unnecessary sql logic

* Adding initial commit of queries_grouped_by_script.sql

* Add a version of queries_grouped_by_hash.sql that retains the raw sql query in case it's useful in future

* aggregate project and reservation ids

* use start_time instead of create_time when scanning history

* aggregate ref_tables as an array instead of comma separated string

* remove unnecessary column

* simplify check for perf insights

* improve query insights scripts and readme

* readme fixes and adding api enabling script

* linting

* add --quiet to bq tool

* Updates

* Update table_read_patterns_by_query_hash.sql

* Update table_read_patterns_by_query_hash.sql

* Update table_read_patterns_by_query_hash.sql

* Update table_read_patterns_by_query_hash.sql

* Update queries_grouped_by_script.sql

* Update queries_grouped_by_script.sql

* Add script queries_grouped_by_labels.sql

* order by total_slot_ms only

* fixes

* add hourly_slot_consumption_by_query_hash.sql

* move file

---------

Co-authored-by: ddeleo <danieldeleo@google.com>
Co-authored-by: Christian Yarros <yarros@google.com>
Co-authored-by: Raj Patel <57837394+rajc242@users.noreply.github.com>

scripts/README.md

@@ -2,4 +2,5 @@
 
 Common script utilities which can help automate common tasks within a BigQuery data warehouse.
 
-* [billing](/scripts/billing) - Example queries over the GCP billing export to help get you started using the standard data export format
+* [billing](/scripts/billing) - Example queries over the GCP billing export to help get you started using the standard data export format.
+* [optimization](/scripts/optimization) - Scripts that identify tables and queries which require optimization.