Import google spreadsheet into bigquery #5
Conversation
As opposed to downloading from s3 bucket
This is implemented as part of the solution for the issue - elifesciences/issues#5256
|
This PR also has code changes relating to Crossref in it. Is that intentional? |
# Conflicts: # dags/crossref_event_data_import_pipeline.py # data_pipeline/crossref_event_data/etl_crossref_event_data_util.py # docker-compose.yml # requirements.txt
@de-code |
|
Hopefully my comments are okay. They are just things I spotted by scanning through the code. It's definitely good overall. Maybe we could discuss ideas to potentially break up PRs in the future. Which may be more difficult for new DAGs but there should be a way. |
|
@de-code - pr based changes made. I am not sure if you want to take another quick look at it before approving it or you are just OK with approving it as it is. |
| def trigger_dag(**kwargs): | ||
| conf_file_path = get_env_var_or_use_default( | ||
| SPREADSHEET_CONFIG_FILE_PATH_ENV_NAME, "" | ||
| def trigger_spreadsheet_data_pipeline_dag(**kwargs): |
There was a problem hiding this comment.
BTW using _ (underscore) will automatically not be flagged as not used
This is implemented as part of the solution for the issue - elifesciences/issues#5256