BUG: ScannerError

[ZeroDot1]

Description

[user@work ~]$ pyfunceble -w 24 -s -f Old.txt 
/home/user/.local/lib/python3.9/site-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.25.11) or chardet (4.0.0) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
The 'cli_testing.file_generation.merge_output_dirs' key is missing from your configuration file.
Are we authorized to merge it from upstream ? [y/n] y

After PyFunceble finished working, these errors occurred.

zpool.ga ACTIVE
Process pyfunceble_dir_files_sorter_worker_11:
Process pyfunceble_dir_files_sorter_worker_10:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/base.py", line 273, in run
    PyFunceble.facility.ConfigLoader.set_custom_config(self.configuration)
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 210, in set_custom_config
    self.custom_config = value
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 136, in wrapper
    self.start()
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 388, in start
    config = self.get_config_file_content()
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 323, in get_config_file_content
    config = self.dict_helper.from_yaml_file(self.path_to_config)
  File "/usr/lib/python3.9/site-packages/PyFunceble/helpers/dict.py", line 291, in from_yaml_file
    data = yaml_load(file_stream)
  File "/usr/lib/python3.9/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python3.9/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/lib/python3.9/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python3.9/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
    if self.check_token(KeyToken):
  File "/usr/lib/python3.9/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/usr/lib/python3.9/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/usr/lib/python3.9/site-packages/yaml/scanner.py", line 577, in fetch_value
    raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
  in "/home/user/.config/PyFunceble/.PyFunceble.yaml", line 147, column 11
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/base.py", line 273, in run
    PyFunceble.facility.ConfigLoader.set_custom_config(self.configuration)
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 210, in set_custom_config
    self.custom_config = value
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 136, in wrapper
    self.start()
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 388, in start
    config = self.get_config_file_content()
  File "/usr/lib/python3.9/site-packages/PyFunceble/config/loader.py", line 323, in get_config_file_content
    config = self.dict_helper.from_yaml_file(self.path_to_config)
  File "/usr/lib/python3.9/site-packages/PyFunceble/helpers/dict.py", line 291, in from_yaml_file
    data = yaml_load(file_stream)
  File "/usr/lib/python3.9/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python3.9/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/lib/python3.9/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/usr/lib/python3.9/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/lib/python3.9/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
    if self.check_token(KeyToken):
  File "/usr/lib/python3.9/site-packages/yaml/scanner.py", line 116, in check_token
    self.fetch_more_tokens()
  File "/usr/lib/python3.9/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/usr/lib/python3.9/site-packages/yaml/scanner.py", line 577, in fetch_value
    raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
  in "/home/user/.config/PyFunceble/.PyFunceble.yaml", line 147, column 11

Execution Time: 00:00:25:5.578765



Status      Percentage   Amount      
----------- ------------ ------------
ACTIVE      92%          2145        
INACTIVE    8%           180         
INVALID     0%           1           
[user@work ~]$ 

[funilrys]

@zerodot, can you send us your configuration file ? It looks like it's not correctly formatted.

[ZeroDot1]

Strange, I changed absolutely nothing in the file, I set all the settings once and then forgot and never touched it again...

Here is the file see link:

https://gist.githubusercontent.com/ZeroDot1/cf4de1e7233318fa6226a52b737b4038/raw/c31980e3f5da815fef97982f05b0077a26e866d6/.PyFunceble.yaml

[ZeroDot1]

[user@work ~]$ pyfunceble -w 24 -s -f Old.txt 
/home/user/.local/lib/python3.9/site-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.25.11) or chardet (4.0.0) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
The 'cli_testing.file_generation.merge_output_dirs' key is missing from your configuration file.
Are we authorized to merge it from upstream ? [y/n] y

OK, after the last update, the configuration file was just completely replaced, and all my settings are gone.
This is not good, and definitely not how it should be.
Unfortunately, I realized that it went so wrong later.

[funilrys]

@ZeroDot1 That's why you should put your own configuration into the .PyFunceble.overwrite.yaml.

For this issue I will invite you to wipe the .PyFunceble.yaml file and put your own settings into .PyFunceble.overwrite.yaml (that file should be generated for you by PyFunceble if it doesn't exists but is never touched again. After the creation PyFunceble will read it and overwrite the standard settings with the one that are in there).

@spirillen may help.

[spirillen]

Just to ensure there is a warning about this 😏

However I'll admit, there should be some kind of confirmation as when .PyFunceble.yaml Y/N that would warn users about this different behavior when they upgrade from 3.x to 4.x where .PyFunceble.yaml is ALWAYS overwritten in 4.x. Hence the new .PyFunceble.overwrite.yaml

@ZeroDot1 do you feel the need for more information about this behavior in the docs?

https://pyfunceble.readthedocs.io/en/dev/configuration/index.html?highlight=.PyFunceble.overwrite.yaml#filename-s (https://mypdns.org/spirillen/PyFunceble/-/blob/4.x-doc/docs/configuration/location.rst#L14)

https://pyfunceble.readthedocs.io/en/dev/usage/index.html#global-variables (https://mypdns.org/spirillen/PyFunceble/-/blob/4.x-doc/docs/usage/bash-terminal.rst#L1251)

[ZeroDot1]

Hi @funilrys @spirillen,

OK, I recreated my configuration file, and put it separately in PyFunceble.overwrite.yaml.
That I had to do it this way was simply not clear to me, because I assumed that I can create a configuration once and then forget it.
It would be better for future changes to display them immediately before the start of PyFunceble or otherwise make users aware of it.
Also, the whole thing should be described exactly in the documentation at the moment it seems a bit unclear to me, because I have nowhere seen the information that the file is simply replaced.
Merging means for me that 2 files are merged to add missing information and not that the entire file is overwritten or deleted.

[spirillen]

@ZeroDot1 is these words "god enough" or do you see room for further improvement?

https://github.com/funilrys/PyFunceble/pull/247/files#diff-d40c282f1ebb52b1000d68a78260a20617e22ff89f83a5ca1916c6c61cff427cR21-R26

[ZeroDot1]

@spirillen OK, perfect, thank you very much.