Add scripts extract and transform the #120
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
"ods_kktix_attendeeId_datetime" table and load to the legecy tables: : ods_kktix_ticket_(corporate, individual, reserved)_attendees
david30907d
reviewed
Jun 20, 2023
contrib/README.md
Outdated
Comment on lines
22
to
27
| ## KKTIX BigQuery Transform | ||
| 1. Background: Start from 2022, we extract the KKTIX data via KKTIX API and load to "pycontw-225217.ods.ods_kktix_attendeeId_datetime". However most of the data are store in the ATTENDEE_INFO column with json format. To use metabase with SQL, users need to extract the data by json_extract with the knowledge kktix format instead of flat database. And we also need to rewrite all the SQLs build for current databases. | ||
| 2. Solution: Transform the tables in backend that we could keep the same user experience by using Metabase. | ||
| 3. Run: | ||
| - `./kktix_bq_etl.py -t ods_kktix_ticket_reserved_attendees_test -k reserved -y 2023 --upload` | ||
| - for 3 tables: `./kktix_bq_etl.py 2023` |
Collaborator
There was a problem hiding this comment.
thx for documenting these up 🙏
david30907d
reviewed
Jun 20, 2023
Comment on lines
54
to
76
| CANONICAL_COLUMN_NAMES_2020_EXTRA_CORPORATE = { | ||
| "invoice_policy", | ||
| "invoiced_company_name", | ||
| "unified_business_no", | ||
| "pynight_attendee_numbers", | ||
| "know_financial_aid", | ||
| "have_you_ever_attended_pycon_tw", | ||
| "pynight_attending_or_not", | ||
| "how_did_you_know_pycon_tw", | ||
| } | ||
|
|
||
| CANONICAL_COLUMN_NAMES_2020_EXTRA_INDIVIDUAL = { | ||
| "pynight_attendee_numbers", | ||
| "know_financial_aid", | ||
| "have_you_ever_attended_pycon_tw", | ||
| "pynight_attending_or_not", | ||
| "how_did_you_know_pycon_tw", | ||
| } | ||
|
|
||
| CANONICAL_COLUMN_NAMES_2020_EXTRA_RESERVED: Set = set() | ||
|
|
||
|
|
||
| CANONICAL_COLUMN_NAMES_2019_CORE = { |
Collaborator
There was a problem hiding this comment.
seems to me that these canonical_xxx sets are unused?
Collaborator
Author
There was a problem hiding this comment.
yes, the main code is from upload-kktix-ticket-csv-to-bigquery.py and I jus keep them, the set was used for unit test.
david30907d
reviewed
Jun 20, 2023
|
|
||
|
|
||
| HEURISTIC_COMPATIBLE_MAPPING_TABLE = { | ||
| # from 2020 reformatted column names |
Collaborator
There was a problem hiding this comment.
thx for writting these comments 🙏
david30907d
reviewed
Jun 20, 2023
| "ive_already_read_and_i_accept_the_epidemic_prevention_of_pycon_tw_2020_pycon_tw_2020_covid19": "ive_already_read_and_i_accept_the_epidemic_prevention_of_pycon_tw", | ||
| "do_you_know_we_have_financial_aid_this_year": "know_financial_aid", | ||
| "contact_email": "email", | ||
| # from 2020 reformatted column names which made it duplicate |
david30907d
reviewed
Jun 20, 2023
contrib/kktix_bq_etl.py
Outdated
Comment on lines
519
to
521
| # print(df.columns) | ||
| sanitized_df = sanitize_column_names(df) | ||
| hash_privacy_info(sanitized_df) |
Collaborator
There was a problem hiding this comment.
thx for remembering hashing these sensitive columns. However, we've already encrypted data on extract side. so no need to do it again~
Collaborator
Author
There was a problem hiding this comment.
yes, could be removed after confirmed.
david30907d
reviewed
Jun 20, 2023
contrib/kktix_bq_etl.py
Outdated
Comment on lines
499
to
500
| df_dict = df_dict.drop(columns = useless_columns) | ||
| df_dict = df_dict.rename(columns = {"reg_no": "registration_no", "ticket_name": "ticket_type", "is_paid": "payment_status"}) |
Collaborator
There was a problem hiding this comment.
what do you thiink if we keep all of the original data in the warehouse, and only deserialize/extract those "useful" columns from JSON to another DB layer?
Collaborator
Author
There was a problem hiding this comment.
Yes, we should keep all the of the original data that maybe useful someday.
…ames in better way
Collaborator
Author
|
Use #123 instead |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Types of changes
Description
Background: Start from 2022, we extract the KKTIX data via KKTIX API and load to "pycontw-225217.ods.ods_kktix_attendeeId_datetime". However most of the data are store in the ATTENDEE_INFO column with json format. To use metabase with SQL, users need to extract the data by json_extract with the knowledge kktix format instead of flat database. And we also need to rewrite all the SQLs build for current databases.
Solution: Transform the tables in backend that we could keep the same user experience by using Metabase.
Checklist:
poetry run pytestlocally to ensure all linter checks passSteps to Test This Pull Request
./kktix_bq_etl.py 2023Expected behavior
The data had been load to ods_kktix_ticket_${ticket_type}_attendees_test on bigquery
ticket_type = corporate individual reserved