Add scripts extract and transform the #120
Conversation
"ods_kktix_attendeeId_datetime" table and load to the legecy tables: : ods_kktix_ticket_(corporate, individual, reserved)_attendees
contrib/README.md
Outdated
| 1. Background: Start from 2022, we extract the KKTIX data via KKTIX API and load to "pycontw-225217.ods.ods_kktix_attendeeId_datetime". However most of the data are store in the ATTENDEE_INFO column with json format. To use metabase with SQL, users need to extract the data by json_extract with the knowledge kktix format instead of flat database. And we also need to rewrite all the SQLs build for current databases. | ||
| 2. Solution: Transform the tables in backend that we could keep the same user experience by using Metabase. | ||
| 3. Run: | ||
| - `./kktix_bq_etl.py -t ods_kktix_ticket_reserved_attendees_test -k reserved -y 2023 --upload` | ||
| - for 3 tables: `./kktix_bq_etl.py 2023` |
There was a problem hiding this comment.
thx for documenting these up 🙏
contrib/kktix_bq_etl.py
Outdated
| CANONICAL_COLUMN_NAMES_2020_EXTRA_CORPORATE = { | ||
| "invoice_policy", | ||
| "invoiced_company_name", | ||
| "unified_business_no", | ||
| "pynight_attendee_numbers", | ||
| "know_financial_aid", | ||
| "have_you_ever_attended_pycon_tw", | ||
| "pynight_attending_or_not", | ||
| "how_did_you_know_pycon_tw", | ||
| } | ||
|
|
||
| CANONICAL_COLUMN_NAMES_2020_EXTRA_INDIVIDUAL = { | ||
| "pynight_attendee_numbers", | ||
| "know_financial_aid", | ||
| "have_you_ever_attended_pycon_tw", | ||
| "pynight_attending_or_not", | ||
| "how_did_you_know_pycon_tw", | ||
| } | ||
|
|
||
| CANONICAL_COLUMN_NAMES_2020_EXTRA_RESERVED: Set = set() | ||
|
|
||
|
|
||
| CANONICAL_COLUMN_NAMES_2019_CORE = { |
There was a problem hiding this comment.
seems to me that these canonical_xxx sets are unused?
There was a problem hiding this comment.
yes, the main code is from upload-kktix-ticket-csv-to-bigquery.py and I jus keep them, the set was used for unit test.
contrib/kktix_bq_etl.py
Outdated
|
|
||
|
|
||
| HEURISTIC_COMPATIBLE_MAPPING_TABLE = { | ||
| # from 2020 reformatted column names |
There was a problem hiding this comment.
thx for writting these comments 🙏
contrib/kktix_bq_etl.py
Outdated
| "ive_already_read_and_i_accept_the_epidemic_prevention_of_pycon_tw_2020_pycon_tw_2020_covid19": "ive_already_read_and_i_accept_the_epidemic_prevention_of_pycon_tw", | ||
| "do_you_know_we_have_financial_aid_this_year": "know_financial_aid", | ||
| "contact_email": "email", | ||
| # from 2020 reformatted column names which made it duplicate |
contrib/kktix_bq_etl.py
Outdated
| # print(df.columns) | ||
| sanitized_df = sanitize_column_names(df) | ||
| hash_privacy_info(sanitized_df) |
There was a problem hiding this comment.
thx for remembering hashing these sensitive columns. However, we've already encrypted data on extract side. so no need to do it again~
There was a problem hiding this comment.
yes, could be removed after confirmed.
contrib/kktix_bq_etl.py
Outdated
| df_dict = df_dict.drop(columns = useless_columns) | ||
| df_dict = df_dict.rename(columns = {"reg_no": "registration_no", "ticket_name": "ticket_type", "is_paid": "payment_status"}) |
There was a problem hiding this comment.
what do you thiink if we keep all of the original data in the warehouse, and only deserialize/extract those "useful" columns from JSON to another DB layer?
There was a problem hiding this comment.
Yes, we should keep all the of the original data that maybe useful someday.
…ames in better way
|
Use #123 instead |
Types of changes
Description
Background: Start from 2022, we extract the KKTIX data via KKTIX API and load to "pycontw-225217.ods.ods_kktix_attendeeId_datetime". However most of the data are store in the ATTENDEE_INFO column with json format. To use metabase with SQL, users need to extract the data by json_extract with the knowledge kktix format instead of flat database. And we also need to rewrite all the SQLs build for current databases.
Solution: Transform the tables in backend that we could keep the same user experience by using Metabase.
Checklist:
poetry run pytestlocally to ensure all linter checks passSteps to Test This Pull Request
./kktix_bq_etl.py 2023Expected behavior
The data had been load to ods_kktix_ticket_${ticket_type}_attendees_test on bigquery
ticket_type = corporate individual reserved