Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add scripts extract and transform the #120

Closed
wants to merge 8 commits into from
Closed

Add scripts extract and transform the #120

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3a5c90b
Add scripts extract and transform the
GeekAngus Jun 18, 2023
79506a8
style: lint the code and fix potential SQL inject by parameterized query
GeekAngus Jun 24, 2023
a3b954c
fix (dataframe): Aggregate the columns with the same purposes or names
GeekAngus Jul 1, 2023
c3064ba
[refactor] (dataframe): Group the columns with the same purposes or n…
GeekAngus Jul 2, 2023
3a2eccb
[refactor] (dataframe): Clean the columns for 2022
GeekAngus Jul 2, 2023
209b5b9
Merge branch 'pycontw:master' into master
GeekAngus Jul 29, 2023
d18249f
[feature] (dag): integrate the script to kktix_loader.py in dag
GeekAngus Aug 12, 2023
cd68ec1
merged
GeekAngus Aug 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion contrib/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,11 @@
3. `python upload-survey-cake-csv-to-bigquery.py --year=<20xx> -c <name of contributor>`
1. it would upload data to Bigquery's `test` dataset
2. If everything looks good, you can `copy` the `fact table` and `dimension table` first
3. Then run `python upload-survey-cake-csv-to-bigquery.py --year=<20xx> -p`. `-p` stands for `production`
3. Then run `python upload-survey-cake-csv-to-bigquery.py --year=<20xx> -p`. `-p` stands for `production`

## KKTIX BigQuery Transform
1. Background: Start from 2022, we extract the KKTIX data via KKTIX API and load to "pycontw-225217.ods.ods_kktix_attendeeId_datetime". However most of the data are store in the ATTENDEE_INFO column with json format. To use metabase with SQL, users need to extract the data by json_extract with the knowledge kktix format instead of flat database. And we also need to rewrite all the SQLs build for current databases.
2. Solution: Transform the tables in backend that we could keep the same user experience by using Metabase.
3. Run:
- `./kktix_bq_etl.py -t ods_kktix_ticket_reserved_attendees_test -k reserved -y 2023 --upload`
- for 3 tables: `./kktix_bq_etl.py 2023`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx for documenting these up 🙏

Loading