BQTag Alpha Version

[ruchirjain86]

First PR for a new tool called as BQTag. Utility class for tagging BQ Table Schemas with Data Catalog Taxonomy Policy Tags. Create BQ Authorised Views using Policy Tags. Helper utility to provision BigQuery Dataset, Data Catalog Taxonomy and Policy Tags.

• Unit tests are present in "tests" directory. Need Data Catalog and BQ API to be enabled in GCP Project.
• Example is present in "example" directory.Need Data Catalog and BQ API to be enabled in GCP Project.
• Readme file is present in the root directory of the tool
• Higher level Readme has been updated with the link to the tool

[suchitpuri]

/gcbrun

[ruchirjain86]

@iht Hi Israel, I would appreciate if you cold review this PR.

[boredabdel]

@iht do you want to take this one ? I can if you don't have time

[boredabdel]

/gcbrun

[iht]

Taking this one for review. Starting now.

[iht]

Please the inline comments for the specific requested changes.

[ruchirjain86]

@iht Thank you Israel for reviewing the code and providing the valuable insights. I have made all the changes suggested by you and pushed it the main branch of my forked repo. Please let me know if any other changes are required.

[iht]

Hi, there are still some Python style bits to be fixed. Perhaps you could try to use some tool such as black to reformat the remaining parts of the code: https://www.freecodecamp.org/news/auto-format-your-python-code-with-black/

[ruchirjain86]

@iht Thank you for the suggestion. I have formatted the code using black and pushed it to my repo.

I also ran the tests to ensure formatting did not break anything and tests ran fine.

(bqtag) ruchirjain@brahmos:~/opensource/bqtag/tools/bqtag/tests$ python3 test_bqtag.py 
2021-07-28 12:27:12,106 DEBUG BQ Client Initialised.
2021-07-28 12:27:12,132 DEBUG Data Catalog Client Initialised.
2021-07-28 12:27:13,330 INFO Taxonomy Created Successfully.
2021-07-28 12:27:16,453 INFO 3 Tags Created Successfully.
2021-07-28 12:27:16,525 DEBUG BQ Client Initialised.
2021-07-28 12:27:16,527 DEBUG Data Catalog Client Initialised.
2021-07-28 12:27:16,528 DEBUG Determining Taxonomy ID using Taxonomy Name provided.
2021-07-28 12:27:17,605 DEBUG Downloading Policy Tags from Taxonomy.
2021-07-28 12:27:18,625 INFO Policy Tags downlaoded from Taxonomy. Total tags: 3
2021-07-28 12:27:18,630 DEBUG BQ Client Initialised.
2021-07-28 12:27:18,632 DEBUG Data Catalog Client Initialised.
2021-07-28 12:27:18,632 DEBUG Determining Taxonomy ID using Taxonomy Name provided.
2021-07-28 12:27:19,718 DEBUG Downloading Policy Tags from Taxonomy.
2021-07-28 12:27:20,718 INFO Policy Tags downlaoded from Taxonomy. Total tags: 3
2021-07-28 12:27:21,440 INFO Dataset created successfully: bqtag-321212.test_dataset.
2021-07-28 12:27:21,996 INFO Table created successfully: bqtag-321212.test_dataset.test_table.
2021-07-28 12:27:22,002 DEBUG BQ Client Initialised.
2021-07-28 12:27:22,005 DEBUG Data Catalog Client Initialised.
2021-07-28 12:27:22,006 DEBUG Determining Taxonomy ID using Taxonomy Name provided.
2021-07-28 12:27:23,091 DEBUG Downloading Policy Tags from Taxonomy.
2021-07-28 12:27:24,092 INFO Policy Tags downlaoded from Taxonomy. Total tags: 3
2021-07-28 12:27:24,092 DEBUG Start Downloading BQ Schema.
2021-07-28 12:27:24,730 INFO Table schema downloaded. Started View Creation.
2021-07-28 12:27:24,730 DEBUG Query Created: SELECT column2, column1, STRUCT(parent.nested2)  as parent FROM `bqtag-321212.test_dataset.test_table`
2021-07-28 12:27:27,502 INFO Authorised view created: bqtag-321212.test_dataset.test_view_medium

.
----------------------------------------------------------------------
Ran 4 tests in 15.401s

OK

[iht]

LGTM

[iht]

/gcbrun