bulk_sync_to_db got UnicodeDecodeError when Chinese characters in dag code

[lvjg]

Apache Airflow version

main (development)

Operating System

centos7

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

What happened

What you expected to happen

dagcode._get_code_from_file open fileloc use encoding=utf-8

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[uranusjr]

You probably need this: https://stackoverflow.com/a/27931669/1376863

Although explicitly using UTF-8 is probably a correct thing to do anyway. IIRC the source code read here is only used for UI display? Always using UTF-8 would be OK if that’s the case.

[potiuk]

Just started to answer when @uranusjr did :) ,. so to repeat - as a solution for now you need to set the encoding properly via LANG* variables in your environment (for all components):
https://stackoverflow.com/questions/2276200/changing-default-encoding-of-python

Airflow will use default encoding from the system. We are working on hard-conding utf8 for other parts #17965 - this might be a nice one to add also for parsing the files.

[lvjg]

thanks~

You probably need this: https://stackoverflow.com/a/27931669/1376863

Although explicitly using UTF-8 is probably a correct thing to do anyway. IIRC the source code read here is only used for UI display? Always using UTF-8 would be OK if that’s the case.

solved...
It is more recommended that we can set the encoding...

[lvjg]

thanks!!!

[Narendra-Neerukonda]

In Airflow 2.1.3, there is an option in core for store_dag_code, which if set to True, stores the dag code in DB and UI retrieves it from there (below snippet from 2.1.3):

def code(cls, fileloc) -> str:
    if STORE_DAG_CODE:
        return cls._get_code_from_db(fileloc)
    else:
        return cls._get_code_from_file(fileloc)

However, in the current main branch, the store_dag_code option seems to have been removed and the code is loaded directly from DB (below snippet from main). So, not sure if the issue will come any more, in future releases.

def code(cls, fileloc) -> str:
    return cls._get_code_from_db(fileloc)

[uranusjr]

I believe Airflow also uses platform encoding when initially reading the source code out to store into the database. This would likely still an issue if that’s the case.

[Narendra-Neerukonda]

I looked into the source code where it's loading dags from files. It's using importlib at airflow.models.dagbag.DagBag:_load_modules_from_file/_load_modules_from_zip, called from airflow.models.dagbag.DagBag:process_file

I'm not aware of the encoding importlib is using internally. But, in the case of this specific issue, user's dag was loaded properly problem faced was to only display the code in the UI.(so assuming it's using utf-8/something-compatible internally referring: PEP-3120
Please let me know if there's any other location in codebase to be looked into.

[potiuk]

Looks lie importlib indeed always uses utf8 unless otherwise specified: https://docs.python.org/3/library/importlib.html#importlib.resources.open_text