Skip to content

Conversation

@okayhooni
Copy link
Contributor

@okayhooni okayhooni commented Feb 27, 2023

BUG FIX FOR THE MR: #27833

1) Handling project location param on async BigQuery dts trigger

Currently, async version of BigQuery dts operator with trigger cannot be used in other regions except us (default) region.
when I used it on another region, trigger task raise error like below.

airflow.exceptions.AirflowException: Trigger failed with exception: 404 The requested transfer config (projects/****/locations/us/transferConfigs/*****) was not found.

I added config param in order to cover all the gcp locations.

2) convert type of field seconds to int, if it was str

requested_run_time is in the template_fields of BigQueryDataTransferServiceStartTransferRunsOperator.
but google python sdk client needs it to be int type. so it must be converted to int type after task is executed.

TypeError: '1676991600' has type str, but expected one of: int

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Feb 27, 2023
@eladkal
Copy link
Contributor

eladkal commented Mar 9, 2023

static tests are failing :(

@okayhooni
Copy link
Contributor Author

static tests are failing :(
@eladkal

I guess this was a just random failure..

[error log]

An unexpected error has occurred: CalledProcessError: command: ('/opt/pipx/venvs/apache-airflow-breeze/bin/python', '-mnodeenv', '--prebuilt', '--clean-src', '/home/runner/.cache/pre-commit/repoui20kj4a/node_env-18.6.0', '-n', '18.6.0')
return code: 1
stdout: (none)
stderr:
     * Install prebuilt node (18.6.0) .Incomplete read while readingfrom https://nodejs.org/download/release/v18.6.0/node-v18.6.0-linux-x64.tar.gz
    .
    Traceback (most recent call last):
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 1519, in <module>
        main()
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 1104, in main
        create_environment(env_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 980, in create_environment
        install_node(env_dir, src_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 739, in install_node
        install_node_wrapped(env_dir, src_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line [76](https://github.com/apache/airflow/actions/runs/4283958493/jobs/7460518056#step:6:77)2, in install_node_wrapped
        download_node_src(node_url, src_dir, args)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 602, in download_node_src
        with ctx as archive:
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/contextlib.py", line 112, in __enter__
        return next(self.gen)
      File "/opt/pipx/venvs/apache-airflow-breeze/lib/python3.7/site-packages/nodeenv.py", line 573, in tarfile_open
        tf = tarfile.open(*args, **kwargs)
      File "/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/tarfile.py", line 1573, in open
        saved_pos = fileobj.tell()
    AttributeError: 'bytes' object has no attribute 'tell'
    
Check the log at /home/runner/.cache/pre-commit/pre-commit.log
There were errors during pre-commit check. They should be fixed
Error: Process completed with exit code 3.

@potiuk potiuk force-pushed the fix/bigquery_dts branch from 90e661d to cdde11a Compare March 12, 2023 18:42
@potiuk
Copy link
Member

potiuk commented Mar 12, 2023

I rebased it. It was a strange error.

@potiuk
Copy link
Member

potiuk commented Mar 14, 2023

@MrGeorgeOwl - can you take a look please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants