Add KKTIX attendee Info ETL to BigQuery pycontw-225217.dwd.kktix_ticket_xxxx_attendees tables

contrib/README.md

@@ -17,4 +17,10 @@
 3. `python upload-survey-cake-csv-to-bigquery.py --year=<20xx> -c <name of contributor>`
     1. it would upload data to Bigquery's `test` dataset
     2. If everything looks good, you can `copy` the `fact table` and `dimension table` first
-    3. Then run `python upload-survey-cake-csv-to-bigquery.py --year=<20xx> -p`. `-p` stands for `production`
+    3. Then run `python upload-survey-cake-csv-to-bigquery.py --year=<20xx> -p`. `-p` stands for `production`
+
+## KKTIX BigQuery Transform
+1. Background: Start from 2022, we extract the KKTIX data via KKTIX API and load to "pycontw-225217.ods.ods_kktix_attendeeId_datetime". However most of the data are store in the ATTENDEE_INFO column with json format. To use metabase with SQL, users need to extract the data by json_extract with the knowledge kktix format instead of flat database. And we also need to rewrite all the SQLs build for current databases.
+2. Solution: Transform the tables in backend that we could keep the same user experience by using Metabase.
+3. Run: 
+ - for 3 tables in single bash script: `./kktix_bq_etl.sh 2023`

contrib/kktix_bq_etl.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+#
+# export GOOGLE_APPLICATION_CREDENTIALS="<where to access service-account.json>"
+#
+project_id="pycontw-225217"
+cmd=${PWD}/../dags/ods/kktix_ticket_orders/udfs/kktix_bq_dwd_etl.py
+
+
+for ticket_type in corporate individual reserved
+do
+    suffix=${ticket_type}_attendees$2
+    cmd_args="-p ${project_id} -d dwd -t kktix_ticket_${suffix} -k ${ticket_type} -y $1 --upload"
+    echo ${cmd_args}
+    ${cmd} ${cmd_args}
+done