_reload_instrumentation_loader sometimes doesn't catch import errors

[BergLucas]

Describe the bug
In the C-modules benchmark, there are several runs where the _reload_instrumentation_loader function does not catch some exceptions when reloading a module, even though there is a try-except:

pynguin/src/pynguin/generator.py

Lines 412 to 415 in 0fbd338

	try:
	importlib.reload(module)
	except Exception as e: # noqa: BLE001
	_LOGGER.warning("Reload of module {module.__name__} failed: %s", e)

However, when I reproduce the run on my PC, the exceptions are caught by the try-except.

Therefore, it might be interesting to investigate why this happens in some cases but not in others. I think it's mainly due to an environment configuration problem, but I'm not sure.

To Reproduce
Steps to reproduce the behaviour:

1. Use Pynguin version '0.41.0'
2. Use the following (minimal) code as a subject for test generation:

Install tensorflow==2.19.0 and target the module tensorflow.python.data.experimental.ops.interleave_ops.

3. Use the following command line arguments to Pynguin:

--algorithm DYNAMOSA --allow_stale_assertions False --any_weight 5 --assertion_generation NONE --bytes_length 20 --call_llm_for_uncovered_targets False --call_llm_on_stall_detection False --change_parameter_probability 0.1 --chop_max_length True --chromosome_length 40 --collection_size 5 --configuration_id subprocess --constant_seeding True --coverage_metrics BRANCH --coverage_threshold 1 --create_coverage_report True --crossover_rate 0.75 --dynamic_constant_seeding True --elite 1 --enable_response_caching False --exploitation_starts_at_percent 0.5 --export_strategy PY_TEST --filter_covered_targets_from_test_cluster False --float_precision 0.01 --focused_config.number_of_mutations 10 --focused_config.number_of_tests_per_target 1 --focused_config.random_test_or_from_archive_probability 0.0 --format_with_black False --hybrid_initial_population False --initial_config.number_of_mutations 1 --initial_config.number_of_tests_per_target 10 --initial_config.random_test_or_from_archive_probability 0.5 --initial_population_mutations 0 --initial_population_seeding False --insertion_uut 0.5 --llm_test_case_percentage 0.5 --max_attempts 1000 --max_delta 20 --max_dynamic_length 1000 --max_dynamic_pool_size 50 --max_initial_tests 10 --max_int 2048 --max_length_test_case 2500 --max_llm_interventions 1 --max_plateau_len 25 --max_recursion 10 --max_sequence_length 10 --max_sequences_combined 10 --max_size 100 --maximum_coverage 100 --maximum_coverage_plateau -1 --maximum_iterations -1 --maximum_memory 3000 --maximum_search_time 600 --maximum_slicing_time 600 --maximum_statement_executions -1 --maximum_test_execution_timeout 5 --maximum_test_executions -1 --min_initial_tests 1 --minimum_coverage 100 --minimum_plateau_iterations -1 --model_name gpt-4o-mini --module_name tensorflow.python.data.experimental.ops.interleave_ops --mutation_order 1 --mutation_strategy FIRST_ORDER_MUTANTS --negate_type 0.1 --none_weight 1 --number_of_mutations 1 --object_reuse_probability 0.9 --original_type_weight 5 --output_path output --output_variables TargetModule ConfigurationId RunId ProjectName TotalTime AlgorithmIterations Coverage RandomSeed CoverageTimeline --population 50 --post_process True --primitive_reuse_probability 0.5 --project_name tensorflow --project_path . --random_perturbation 0.2 --rank_bias 1.7 --report_dir output --run_id 5220-16 --seed 26 --seed_from_archive False --seed_from_archive_mutations 3 --seed_from_archive_probability 0.2 --seeded_dynamic_values_reuse_probability 0.6 --seeded_primitives_reuse_probability 0.2 --seeded_testcases_reuse_probability 0.9 --selection TOURNAMENT_SELECTION --skip_optional_parameter_probability 0.7 --statement_insertion_probability 0.5 --statistics_backend CSV --string_length 20 --temperature 0.8 --test_change_probability 0.3333333333333333 --test_delete_probability 0.3333333333333333 --test_execution_time_per_statement 1 --test_insert_probability 0.3333333333333333 --test_insertion_probability 0.1 --timeline_interpolation True --timeline_interval 1000000000 --tournament_size 5 --type4py_weight 10 --type_guess_top_n 10 --type_inference_strategy TYPE_HINTS --type_tracing 0.0 --type_tracing_kept_guesses 2 --type_tracing_weight 10 --use_archive False --use_random_object_for_call 0.1 --wrap_var_param_type_probability 0.7 -v --subprocess

4. Give the error (stack trace, etc) you are encountering:

[08:27:26] [Info] (pynguin.generator:run_pynguin:127): Stop Pynguin Test Generation…
Traceback (most recent call last):
  File "/usr/local/bin/pynguin", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/pynguin/cli.py", line 213, in main
    return run_pynguin().value
  File "/usr/local/lib/python3.10/site-packages/pynguin/generator.py", line 125, in run_pynguin
    return _run()
  File "/usr/local/lib/python3.10/site-packages/pynguin/generator.py", line 545, in _run
    tracked_metrics = _track_final_metrics(
  File "/usr/local/lib/python3.10/site-packages/pynguin/generator.py", line 446, in _track_final_metrics
    _reload_instrumentation_loader(
  File "/usr/local/lib/python3.10/site-packages/pynguin/generator.py", line 382, in _reload_instrumentation_loader
    importlib.reload(module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 169, in reload
    _bootstrap._exec(spec, module)
  File "<frozen importlib._bootstrap>", line 619, in _exec
  File "/usr/local/lib/python3.10/site-packages/pynguin/instrumentation/machinery.py", line 60, in exec_module
    super().exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/local/lib/python3.10/site-packages/tensorflow/python/data/experimental/ops/interleave_ops.py", line 261, in <module>
    v2_compat.register_data_v2_callback(_tf2_callback)
  File "/usr/local/lib/python3.10/site-packages/tensorflow/python/compat/v2_compat.py", line 36, in register_data_v2_callback
    _DATA_V2_CALLBACKS.register(data_v2_func, data_v2_func.__module__)
  File "/usr/local/lib/python3.10/site-packages/tensorflow/python/framework/registry.py", line 57, in register
    raise KeyError(
KeyError: "Registering two data_v2_callbacks with name 'tensorflow.python.data.experimental.ops.interleave_ops'! (Previous registration was in register /usr/local/lib/python3.10/site-packages/tensorflow/python/framework/registry.py:65)"
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown.

from the C-modules benchmark run 5220-16

Expected behavior

INFO     Stopping condition reached                                                                                                                                                                                         generator.py:565
INFO     Used search time: 601/600                                                                                                                                                                                          generator.py:567
INFO     Used memory: 1891.84375/3000.0 MB                                                                                                                                                                                  generator.py:567
INFO     Stop generating test cases                                                                                                                                                                                         generator.py:568
WARNING  Reload of module {module.__name__} failed: "Registering two data_v2_callbacks with name 'tensorflow.python.data.experimental.ops.interleave_ops'! (Previous registration was in register                           generator.py:415
        /home/lucas/.conda/envs/pynguin/lib/python3.10/site-packages/tensorflow/python/framework/registry.py:65)"                                                                                                         
INFO     Calculating resulting FinalBranchCoverage                                                                                                                                                                          generator.py:489
INFO     Written 6 test cases to /output/test_tensorflow_python_data_experimental_ops_interleave_ops.py                                                                                                                     generator.py:832
INFO     Writing statistics                                                                                                                                                                                                     stats.py:363
INFO     Stop Pynguin Test Generation…                                                                                                                                                                                      generator.py:129

Software Version (please complete the following information):

• OS: Fedora 42
• Python version: 3.10
• Pynguin Version: 0.41.0

Additional info
There's also a bug in the call to the log function:

pynguin/src/pynguin/generator.py

Line 415 in 0fbd338

_LOGGER.warning("Reload of module {module.__name__} failed: %s", e)

The string is supposed to be an f-string to format the module name in the string, but since it isn't, it just displays {module.__name__}.

[LuKrO2011]

Hi @BergLucas,

thank you for reporting this Bug. I can confirm it, and it happens on other modules as well. It seems to only happen with modules that are not meant for being reloaded.

Regarding the log call-f-string-bug: I will fix that right away.

[LuKrO2011]

Could it be, that the error is caught and logged correctly and something after the reloading actually causes a crash?

[BergLucas]

I was thinking about that too, but since there's no print of Reload of module {module.__name__} failed:, it seems unlikely to me.

[LuKrO2011]

Hmm, true. I did a bit of googling and found a couple of instances where except Exception does not work. These include:

• If the exception class does not inherit from Exception — but KeyError does
• If the exception is raised in a sub-process or different thread — but then the Error should not make Pynguin crash

However, I don't see how any of these cases matches the described issue.

[BergLucas]

Yeah, and it's strange that it crashed on the benchmark server but not on my PC, even though the code hasn't changed.

[stephanlukasczyk]

Random thought: do the benchmark server and your PC use the exact same Python version, @BergLucas ? A bugfix in Python could cause the different behaviour, too.

And, by the way, the most generic exception in Python is BaseException, "In Python, all exceptions must be instances of a class that derives from [...] it". While that does not explain the above, it's still worth noting that at some point one might want to actually catch BaseException instead of Exception.

[BergLucas]

I was using Python 3.10.14 in my case.

[LuKrO2011]

On the cluster, 3.10.16 was used as defined in the Dockerfile.

[stephanlukasczyk]

Could you try with both versions? Adjusting the version in Docker is easy, don't know about your machine, though. Just that single example module, executed manually.

[BergLucas]

I've just tested on my PC with Python version 3.10.16, and the exception is caught just like with Python version 3.10.14.

[BergLucas]

I think I found a similar case in run 2046-28:

[13:22:37] [Info] (pynguin.generator:run_pynguin:127): Stop Pynguin Test Generation…
Traceback (most recent call last):
  File "/usr/local/bin/pynguin", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/pynguin/cli.py", line 213, in main
    return run_pynguin().value
  File "/usr/local/lib/python3.10/site-packages/pynguin/generator.py", line 125, in run_pynguin
    return _run()
  File "/usr/local/lib/python3.10/site-packages/pynguin/generator.py", line 554, in _run
    coverage_report = get_coverage_report(
  File "/usr/local/lib/python3.10/site-packages/pynguin/utils/report.py", line 162, in get_coverage_report
    source = inspect.getsourcelines(sys.modules[config.configuration.module_name])[0]
  File "/usr/local/lib/python3.10/inspect.py", line 1121, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/local/lib/python3.10/inspect.py", line 940, in findsource
    file = getsourcefile(object)
  File "/usr/local/lib/python3.10/inspect.py", line 817, in getsourcefile
    filename = getfile(object)
  File "/usr/local/lib/python3.10/inspect.py", line 797, in getfile
    raise TypeError('module, class, method, function, traceback, frame, or '
TypeError: module, class, method, function, traceback, frame, or code object was expected, got _Block

pynguin/src/pynguin/utils/report.py

Lines 162 to 168 in 16b4676

	try:
	source = inspect.getsourcelines(sys.modules[config.configuration.module_name])[0]
	except Exception as e:
	# circular references in module might cause errors
	raise RuntimeError(
	f"Could not get source code for module {config.configuration.module_name}. "
	) from e

[LuKrO2011]

No, that's a different case: I updated the code for this after the experiment runs in 74ce24c.

[BergLucas]

Sorry, I didn't see that the try-except came from one of your fixes.