Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logs print credentials if error in catalog with Rich Logging show_locals=True #1712

Closed
Debbby57 opened this issue Jul 19, 2022 · 3 comments · Fixed by #1726
Closed

logs print credentials if error in catalog with Rich Logging show_locals=True #1712

Debbby57 opened this issue Jul 19, 2022 · 3 comments · Fixed by #1726
Assignees

Comments

@Debbby57
Copy link

Debbby57 commented Jul 19, 2022

Description

I run a pipeline but I made a mistake with a credentials key for fsspec (I used HOST instead of host ). Kedro crashes (I agree) but in the logs, all my credentials have been printed. Those credentials for this dataset are printed multiple times but all my credentials were printed too.

Context

Steps to Reproduce

  1. create a credentials dictionnary to connect with a sftp connection
  2. make a mistake in a key (for example : HOST instead of `host)
  3. create a dataset in the catalog with needs thhose credentials
  4. run the pipeline

Expected Result

the log should hide credentials.

Actual Result

the log prints all credentials.

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│  │
│ /site-packages/kedro/io/core.py:155 in from_config                                               │
│                                                                                                  │
│   152 │   │   │   ) from exc                                                                     │
│   153 │   │                                                                                      │
│   154 │   │   try:                                                                               │
│ ❱ 155 │   │   │   data_set = class_obj(**config)  # type: ignore                                 │
│   156 │   │   except TypeError as err:                                                           │
│   157 │   │   │   raise DataSetError(                                                            │
│   158 │   │   │   │   f"\n{err}.\nDataSet '{name}' must only contain arguments valid for the "   │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │    class_obj = <class 'kedro.extras.datasets.pandas.csv_dataset.CSVDataSet'>                 │ │
│ │          cls = <class 'kedro.io.core.AbstractDataSet'>                                       │ │
│ │       config = {                                                                             │ │
│ │                │   'filepath':                                                               │ │
│ │                'sftp://my_path/s… │ │
│ │                │   'save_args': {'sep': ';'},                                                │ │
│ │                │   'credentials': {                                                          │ │
│ │                │   │   'username': '',                                                │ │
│ │                │   │   'password': '',                                              │ │
│ │                │   │   'HOST': '',                                             │ │
│ │                │   │   'port': ''                                                          │ │
│ │                │   }                                                                         │ │
│ │                }                                                                             │ │
│ │ load_version = None                                                                          │ │
│ │         name = 'my_data'                                                              │ │
│ │ save_version = '2022-07-19T08.14.05.166Z'                                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│  │
│ /site-packages/kedro/extras/datasets/pandas/csv_dataset.py:118 in __init__                       │
│                                                                                                  │
│   115 │   │                                                                                      │
│   116 │   │   self._protocol = protocol                                                          │
│   117 │   │   self._storage_options = {**_credentials, **_fs_args}                               │
│ ❱ 118 │   │   self._fs = fsspec.filesystem(self._protocol, **self._storage_options)              │
│   119 │   │                                                                                      │
│   120 │   │   super().__init__(                                                                  │
│   121 │   │   │   filepath=PurePosixPath(path),                                                  │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │    __class__ = <class 'kedro.extras.datasets.pandas.csv_dataset.CSVDataSet'>                 │ │
│ │ _credentials = {                                                                             │ │
│ │                │   'username': '',                                                    │ │
│ │                │   'password': '',                                                  │ │
│ │                │   'HOST': '',                                                 │ │
│ │                │   'port': ''                                                              │ │
│ │                }                                                                             │ │
│ │     _fs_args = {}                                                                            │ │
│ │  credentials = {                                                                             │ │
│ │                │   'username': '',                                                    │ │
│ │                │   'password': '',                                                  │ │
│ │                │   'HOST': '',                                                 │ │
│ │                │   'port': ''                                                              │ │
│ │                }                                                                             │ │
│ │     filepath = 'sftp://my_path/s… │ │
│ │      fs_args = None                                                                          │ │
│ │    load_args = None                                                                          │ │
│ │         path = 'path… │ │
│ │     protocol = 'sftp'                                                                        │ │
│ │    save_args = {'sep': ';'}                                                                  │ │
│ │         self = <kedro.extras.datasets.pandas.csv_dataset.CSVDataSet object at                │ │
│ │                0x7f81debdb400>                                                               │ │
│ │      version = None                                                                          │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│  │
│ /site-packages/fsspec/registry.py:253 in filesystem                                              │
│                                                                                                  │
│   250 │   passed directly to the class.                                                          │
│   251 │   """                                                                                    │
│   252 │   cls = get_filesystem_class(protocol)                                                   │
│ ❱ 253 │   return cls(**storage_options)                                                          │
│   254                                                                                            │
│                                                                                                  │
│ ╭──────────────────────────────── locals ────────────────────────────────╮                       │
│ │             cls = <class 'fsspec.implementations.sftp.SFTPFileSystem'> │                       │
│ │        protocol = 'sftp'                                               │                       │
│ │ storage_options = {                                                    │                       │
│ │                   │   'username': '',                           │                       │
│ │                   │   'password': '',                         │                       │
│ │                   │   'HOST': '',                        │                       │
│ │                   │   'port': ''                                     │                       │
│ │                   }                                                    │                       │
│ ╰────────────────────────────────────────────────────────────────────────╯                       │
│                                                                                                  │
│  │
│ /site-packages/fsspec/spec.py:68 in __call__                                                     │
│                                                                                                  │
│     65 │   │   if not skip and cls.cachable and token in cls._cache:                             │
│     66 │   │   │   return cls._cache[token]                                                      │
│     67 │   │   else:                                                                             │
│ ❱   68 │   │   │   obj = super().__call__(*args, **kwargs)                                       │
│     69 │   │   │   # Setting _fs_token here causes some static linters to complain.              │
│     70 │   │   │   obj._fs_token_ = token                                                        │
│     71 │   │   │   obj.storage_args = args                                                       │
│                                                                                                  │
│ ╭────────────────────────────── locals ───────────────────────────────╮                          │
│ │    __class__ = <class 'fsspec.spec._Cached'>                        │                          │
│ │         args = ()                                                   │                          │
│ │          cls = <class 'fsspec.implementations.sftp.SFTPFileSystem'> │                          │
│ │ extra_tokens = ()                                                   │                          │
│ │       kwargs = {                                                    │                          │
│ │                │   'username': '',                           │                          │
│ │                │   'password': '',                         │                          │
│ │                │   'HOST': '',                        │                          │
│ │                │   'port': ''                                     │                          │
│ │                }                                                    │                          │
│ │         skip = False                                                │                          │
│ │        token = '46ce48dd0c14bdb4185ae63058731b3b'                   │                          │
│ ╰─────────────────────────────────────────────────────────────────────╯                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: __init__() missing 1 required positional argument: 'host'

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ //env/bin/kedro:8   │
│ in <module>                                                                                      │
│                                                                                                  │
│  │
│ /site-packages/kedro/framework/cli/cli.py:211 in main                                            │
│                                                                                                  │
│   208 │   """                                                                                    │
│   209 │   _init_plugins()                                                                        │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                     │
│ ❱ 211 │   cli_collection()                                                                       │
│   212                                                                                            │
│                                                                                                  │
│ ╭───────────── locals ─────────────╮                                                             │
│ │ cli_collection = <KedroCLI None> │                                                             │
│ ╰──────────────────────────────────╯                                                             │
│                                                                                                  │
│  │
│ /site-packages/click/core.py:1130 in __call__                                                    │
│                                                                                                  │
│  │
│ /site-packages/kedro/framework/cli/cli.py:139 in main                                            │
│                                                                                                  │
│   136 │   │   )                                                                                  │
│   137 │   │                                                                                      │
│   138 │   │   try:                                                                               │
│ ❱ 139 │   │   │   super().main(                                                                  │
│   140 │   │   │   │   args=args,                                                                 │
│   141 │   │   │   │   prog_name=prog_name,                                                       │
│   142 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │       __class__ = <class 'kedro.framework.cli.cli.KedroCLI'>                                 │ │
│ │            args = ['run']                                                                    │ │
│ │    complete_var = None                                                                       │ │
│ │           extra = {                                                                          │ │
│ │                   │   'obj': ProjectMetadata(                                                │ │
│ │                   │   │                                                                      │ │
│ │                   config_file=PosixPath('/path… │ │
│ │                   │   │   package_name='package_name',                                │ │
│ │                   │   │   project_name='project_name',                                │ │
│ │                   │   │                                                                      │ │
│ │                   project_path=PosixPath('/path… │ │
│ │                   │   │   project_version='0.18.1',                                          │ │
│ │                   │   │                                                                      │ │
│ │                   source_dir=PosixPath('/path… │ │
│ │                   │   )                                                                      │ │
│ │                   }                                                                          │ │
│ │       prog_name = None                                                                       │ │
│ │            self = <KedroCLI None>                                                            │ │
│ │ standalone_mode = True                                                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│  │
│ /site-packages/click/core.py:1055 in main                                                        │
│                                                                                                  │
│  │
│ /site-packages/click/core.py:1657 in invoke                                                      │
│                                                                                                  │
│  │
│ /site-packages/click/core.py:1404 in invoke                                                      │
│                                                                                                  │
│  │
│ /site-packages/click/core.py:760 in invoke                                                       │
│                                                                                                  │
│  │
│ /site-packages/framework_cli/cli.py:111 in run                                                   │
│                                                                                                  │
│   108 │   │   │   else get_globals_entry(config_loader._config_mapping, "runner")                │
│   109 │   │   )                                                                                  │
│   110 │   │   runner_obj = init_runner(runner, is_async, config_loader)                          │
│ ❱ 111 │   │   session.run(                                                                       │
│   112 │   │   │   tags=tag,                                                                      │
│   113 │   │   │   runner=runner_obj,                                                             │
│   114 │   │   │   node_names=node_names,                                                         │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │        config = None                                                                         │ │
│ │ config_loader = <custom_config_loader.CustomConfigLoader object   │ │
│ │                 at 0x7f820241c760>                                                           │ │
│ │           env = None                                                                         │ │
│ │   from_inputs = []                                                                           │ │
│ │    from_nodes = []                                                                           │ │
│ │      is_async = False                                                                        │ │
│ │  load_version = {}                                                                           │ │
│ │    node_names = ()                                                                           │ │
│ │        params = {}                                                                           │ │
│ │       pipeine = None                                                                         │ │
│ │      pipeline = None                                                                         │ │
│ │        runner = None                                                                         │ │
│ │    runner_obj = <kedro.runner.sequential_runner.SequentialRunner object at 0x7f82007fd9a0>   │ │
│ │       session = <kedro.framework.session.session.KedroSession object at 0x7f8200a40250>      │ │
│ │           tag = ()                                                                           │ │
│ │      to_nodes = []                                                                           │ │
│ │    to_outputs = []                                                                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│  │
│ /site-packages/kedro/framework/session/session.py:389 in run                                     │
│                                                                                                  │
│   386 │   │   │   "runner": getattr(runner, "__name__", str(runner)),                            │
│   387 │   │   }                                                                                  │
│   388 │   │                                                                                      │
│ ❱ 389 │   │   catalog = context._get_catalog(                                                    │
│   390 │   │   │   save_version=save_version,                                                     │
│   391 │   │   │   load_versions=load_versions,                                                   │
│   392 │   │   )                                                                                  │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │           context = <kedro.framework.context.context.KedroContext object at 0x7f82007ea730>  │ │
│ │      extra_params = {}                                                                       │ │
│ │ filtered_pipeline = Pipeline([                                                               │ │
│ │                     Node(extract_df, ['ee'], ['dd'], None),               │ │
│ │                     Node(change_var_format, ['a'], 'my_data', None)         │ │
│ │                     ])                                                                       │ │
│ │       from_inputs = []                                                                       │ │
│ │        from_nodes = []                                                                       │ │
│ │     load_versions = {}                                                                       │ │
│ │              name = '__default__'                                                            │ │
│ │        node_names = ()                                                                       │ │
│ │          pipeline = Pipeline([                                                               │ │
│ │                     Node(extract_df, ['extract_stop'], ['a'], None),               │ │
│ │                     Node(change_var_format, ['a'], 'my_data', None)         │ │
│ │                     ])                                                                       │ │
│ │     pipeline_name = None                                                                     │ │
│ │       record_data = {                                                                        │ │
│ │                     │   'session_id': '2022-07-19T08.14.05.166Z',                            │ │
│ │                     │   'project_path':                                                      │ │
│ │                     '/path… │ │
│ │                     │   'env': None,                                                         │ │
│ │                     │   'kedro_version': '0.18.2',                                           │ │
│ │                     │   'tags': (),                                                          │ │
│ │                     │   'from_nodes': [],                                                    │ │
│ │                     │   'to_nodes': [],                                                      │ │
│ │                     │   'node_names': (),                                                    │ │
│ │                     │   'from_inputs': [],                                                   │ │
│ │                     │   'to_outputs': [],                                                    │ │
│ │                     │   ... +4                                                               │ │
│ │                     }                                                                        │ │
│ │            runner = <kedro.runner.sequential_runner.SequentialRunner object at               │ │
│ │                     0x7f82007fd9a0>                                                          │ │
│ │      save_version = '2022-07-19T08.14.05.166Z'                                               │ │
│ │              self = <kedro.framework.session.session.KedroSession object at 0x7f8200a40250>  │ │
│ │        session_id = '2022-07-19T08.14.05.166Z'                                               │ │
│ │              tags = ()                                                                       │ │
│ │          to_nodes = []                                                                       │ │
│ │        to_outputs = []                                                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│  │
│ /site-packages/kedro/framework/context/context.py:286 in _get_catalog                            │
│                                                                                                  │
│   283 │   │   )                                                                                  │
│   284 │   │   conf_creds = self._get_config_credentials()                                        │
│   285 │   │                                                                                      │
│ ❱ 286 │   │   catalog = settings.DATA_CATALOG_CLASS.from_config(                                 │
│   287 │   │   │   catalog=conf_catalog,                                                          │
│   288 │   │   │   credentials=conf_creds,                                                        │
│   289 │   │   │   load_versions=load_versions,                                                   │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │  conf_catalog = {                                                                            │ │
│ │                 │   'extract_stop': {                                                        │ │
│ │                 │   │   'type': 'custom_dataset',                               			 │
│ │                 │   │   'custom_query': "xxxxxxx"+129,                    					 │ │
│ │                 │   │   'parameters': {                                                      │ │
│ │                 │   │   │   'programs_path':                                                 │ │
│ │                 'some_path',           														 │ │
│ │                 │   │   │   'date': '2022-06-30',                                            │ │
																								 │ │
│ │                 │   │   },                                                                   │ │
│ │                 │   },                                                                       │ │
                                                                      │ │
│ │                 │   'my_data': {                                                      │ │
│ │                 │   │   'type': 'pandas.CSVDataSet',                                         │ │
│ │                 │   │   'filepath':                                                          │ │
│ │                 'sftp://my_path/… │ │
│ │                 │   │   'save_args': {'sep': ';'},                                           │ │
│ │                 │   │   'credentials': 'test_creds'                                          │ │
│ │                 │   }                                                                        │ │
│ │                 }                                                                            │ │
│ │    conf_creds = {                                                                            │ │
│ │                 │   'test_creds': {                                                          │ │
│ │                 │   │   'username': '',                                               │ │
│ │                 │   │   'password': '',                                             │ │
│ │                 │   │   'HOST': ',                                            │ │
│ │                 │   │   'port': ''                                                         │ │
│ │                 │   },                                                                       │ │
│ │                 │   'other_creds': {                                                       │ │
│ │                 │   │   'aa': '',                                            │ │
│ │                 │   │   'bb': '',                                          │ │
│ │                 │   │   'cc': '',                               │ │
│ │                 │   │   'dd': ''                                                    │ │
│ │                 │   },                                                                       │ │
│ │                 │   'local_postgre_creds': {                                                 │ │
│ │                 │   │   'con':                                                               │ │
│ │                 'postgresql:
│ │                 │   }                                                                        │ │
│ │                 }                                                                            │ │
│ │ load_versions = {}                                                                           │ │
│ │  save_version = '2022-07-19T08.14.05.166Z'                                                   │ │
│ │          self = <kedro.framework.context.context.KedroContext object at 0x7f82007ea730>      │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│  │
│ /site-packages/kedro/io/data_catalog.py:277 in from_config                                       │
│                                                                                                  │
│   274 │   │   │   │   layers[ds_layer].add(ds_name)                                              │
│   275 │   │   │                                                                                  │
│   276 │   │   │   ds_config = _resolve_credentials(ds_config, credentials)                       │
│ ❱ 277 │   │   │   data_sets[ds_name] = AbstractDataSet.from_config(                              │
│   278 │   │   │   │   ds_name, ds_config, load_versions.get(ds_name), save_version               │
│   279 │   │   │   )                                                                              │
│   280                                                                                            │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │       catalog = {                                                                            │ │
│ │                 │   'extract_stop': {                                                        │ │
│ │                 │   │   'type': 'custom_dataset',                               			 │
│ │                 │   │   'custom_query': "xxxxxxx"+129,                    					 │ │
│ │                 │   │   'parameters': {                                                      │ │
│ │                 │   │   │   'programs_path':                                                 │ │
│ │                 'some_path',           														 │ │
│ │                 │   │   │   'date': '2022-06-30',                                            │ │
																								 │ │
│ │                 │   │   },                                                                   │ │
│ │                 │   },                                                                       │ │
                                                                      │ │
│ │                 │   'my_data': {                                                      │ │
│ │                 │   │   'type': 'pandas.CSVDataSet',                                         │ │
│ │                 │   │   'filepath':                                                          │ │
│ │                 'sftp://my_path/… │ │
│ │                 │   │   'save_args': {'sep': ';'},                                           │ │
│ │                 │   │   'credentials': 'test_creds'                                          │ │
│ │                 │   }                                                                        │ │
│ │                 }                                                                            │ │
│ │           cls = <class 'kedro.io.data_catalog.DataCatalog'>                                  │ │
│ │   credentials = {                                                                            │ │
│ │                 │   'test_creds': {                                                          │ │
│ │                 │   │   'username': '',                                               │ │
│ │                 │   │   'password': '',                                             │ │
│ │                 │   │   'HOST': '',                                            │ │
│ │                 │   │   'port': ''                                                         │ │
│ │                 │   },                                                                       │ │
│ │                 │   'other_creds': {                                                       │ │
│ │                 │   │   'aa': '',                                            │ │
│ │                 │   │   'bb': '',                                          │ │
│ │                 │   │   'cc': '',                               │ │
│ │                 │   │   'dd': ''                                                    │ │
│ │                 │   },                                                                       │ │
																								 │ │
│ │                 │   'local_postgre_creds': {                                                 │ │
│ │                 │   │   'con':                                                               │ │
│ │                 'postgresql://            │ │
│ │                 │   }                                                                        │ │
│ │                 }                                                                            │ │
│ │     data_sets = {                                                                            │ │
│ │                 │   'extract_stop': <custom_dataset object   │ │
│ │                 at 0x7f82007fde20>,                                                          │ │
│ │                 │   'a': <custom_dataset object at │ │
│ │                 0x7f81dfb08a30>                                                              │ │
│ │                 }                                                                            │ │
│ │     ds_config = {                                                                            │ │
│ │                 │   'type': 'pandas.CSVDataSet',                                             │ │
│ │                 │   'filepath':                                                              │ │
│ │                 'sftp://my_path/… │ │
│ │                 │   'save_args': {'sep': ';'},                                               │ │
│ │                 │   'credentials': {                                                         │ │
│ │                 │   │   'username': '',                                               │ │
│ │                 │   │   'password': '',                                             │ │
│ │                 │   │   'HOST': '',                                            │ │
│ │                 │   │   'port': ''                                                         │ │
│ │                 │   }                                                                        │ │
│ │                 }                                                                            │ │
│ │      ds_layer = None                                                                         │ │
│ │       ds_name = 'my_data'                                                             │ │
│ │        layers = defaultdict(<class 'set'>, {})                                               │ │
│ │ load_versions = {}                                                                           │ │
│ │  missing_keys = set()                                                                        │ │
│ │  save_version = '2022-07-19T08.14.05.166Z'                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│  │
│ /site-packages/kedro/io/core.py:157 in from_config                                               │
│                                                                                                  │
│   154 │   │   try:                                                                               │
│   155 │   │   │   data_set = class_obj(**config)  # type: ignore                                 │
│   156 │   │   except TypeError as err:                                                           │
│ ❱ 157 │   │   │   raise DataSetError(                                                            │
│   158 │   │   │   │   f"\n{err}.\nDataSet '{name}' must only contain arguments valid for the "   │
│   159 │   │   │   │   f"constructor of '{class_obj.__module__}.{class_obj.__qualname__}'."       │
│   160 │   │   │   ) from err                                                                     │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │    class_obj = <class 'kedro.extras.datasets.pandas.csv_dataset.CSVDataSet'>                 │ │
│ │          cls = <class 'kedro.io.core.AbstractDataSet'>                                       │ │
│ │       config = {                                                                             │ │
│ │                │   'filepath':                                                               │ │
│ │                'sftp://my_path/s… │ │
│ │                │   'save_args': {'sep': ';'},                                                │ │
│ │                │   'credentials': {                                                          │ │
│ │                │   │   'username': '',                                                │ │
│ │                │   │   'password': '',                                              │ │
│ │                │   │   'HOST': '',                                             │ │
│ │                │   │   'port': ''                                                          │ │
│ │                │   }                                                                         │ │
│ │                }                                                                             │ │
│ │ load_version = None                                                                          │ │
│ │         name = 'my_data'                                                              │ │
│ │ save_version = '2022-07-19T08.14.05.166Z'                                                    │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
DataSetError: 
__init__() missing 1 required positional argument: 'host'.
DataSet 'my_data' must only contain arguments valid for the constructor of 'kedro.extras.datasets.pandas.csv_dataset.CSVDataSet'.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

  • Kedro version used : project created in 0.18.1 but I run the pipeline with the new version kedro 0.18.2
  • Python version used (python -V): 3.9.13
  • Operating system and version: linux
@Debbby57 Debbby57 changed the title logs print credentials logs print credentials if error in catalog Jul 19, 2022
@datajoely
Copy link
Contributor

Thank you @Debbby57 for raising this - we'll look into how we can fix this.

@antonymilne
Copy link
Contributor

antonymilne commented Jul 19, 2022

Hi @Debbby57, thanks very much for raising this. Just to check: is the only place that credentials appear in the boxes showing locals?

This is a side-effect of us recently (in 0.18.2) switching to using rich tracebacks for exception handling. We hadn't thought that this would cause problems like you've seen here, sorry! Just for now, a quick fix would be to add the following lines to your project settings.py file:

import rich.traceback
import click
from pathlib import Path
import sys

rich.traceback.install(
    show_locals=False, suppress=[click, str(Path(sys.executable).parent)]
)

This will keep the same rich exception handling but no longer show the local variables (since show_locals=False).

Longer term, I think the only fix here would be to set show_locals=False in the kedro framework. This seems like a bit of a pity since I think it's a feature many people find handy, but there's no way of hiding particular local variables in rich so it's all or nothing here. I'd be interested to hear what other people think about how serious the issue raised here is, but ultimately I think we'll have to prioritise security over convenience here.

The alternative is we try and raise a PR in rich to add some sort of hide_locals option or the like that would enable us to not show credentials. But it seems like a bit of a weird feature to add tbh so no idea if they would want it.

@Debbby57
Copy link
Author

thank you very much for your reply ! The fix you propose works fine :)

I confirm that my credentials only appear in the "locals" boxes.

From a safety point of view, I am really not comfortable to print credentials by default.
It's a really good feature because when we debug we have a tendency to print local variables to understand where the bug is, but as I am concerned, I prefer to debug manually and choose what I print in my logs instead of having my credentials in logs by default.

@noklam noklam changed the title logs print credentials if error in catalog logs print credentials if error in catalog with Rich Logging show_locals=True Jul 21, 2022
@merelcht merelcht self-assigned this Jul 26, 2022
@merelcht merelcht linked a pull request Jul 26, 2022 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants