Running lstchain_mc_r0_to_dl1 with default gamma_test_large.simtel.gz fails

[morcuended]

Running lstchain_mc_r0_to_dl1 with v0.10.11 and gave this error:

Downloading gamma_test_large.simtel.gz ...

Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/lstchain-v0.10.11/bin/lstchain_mc_r0_to_dl1", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/fefs/aswg/software/conda/envs/lstchain-v0.10.11/lib/python3.11/site-packages/lstchain/scripts/lstchain_mc_r0_to_dl1.py", line 74, in main
    r0_to_dl1.r0_to_dl1(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.10.11/lib/python3.11/site-packages/lstchain/reco/r0_to_dl1.py", line 352, in r0_to_dl1
    source = EventSource(input_url=input_filename,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fefs/aswg/software/conda/envs/lstchain-v0.10.11/lib/python3.11/site-packages/ctapipe/io/simteleventsource.py", line 512, in __init__
    self._subarray_info = self.prepare_subarray_info(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fefs/aswg/software/conda/envs/lstchain-v0.10.11/lib/python3.11/site-packages/ctapipe/io/simteleventsource.py", line 643, in prepare_subarray_info
    raise RuntimeError(
RuntimeError: `SimTelEventSource.focal_length_choice` was set to 'EFFECTIVE', but the effective focal length was not present in the file. Set `focal_length_choice='EQUIVALENT'` or make sure input files contain the effective focal length

[maxnoe]

Could be fixed by either

• Requiring an inputfile and not using the old prod3 file as default
• Changing the default file to gamma_prod5.simtel.zst which does have the effective focal length

[morcuended]

With this file gamma_prod5.simtel.zst I got No events in file

[maxnoe]

There might be no LST 1 events in there, I guess the default is aalowedtels=[1]?

[morcuended]

There might be no LST 1 events in there, I guess the default is aalowedtels=[1]?

yes, then I suggest to require the user explicitly an input file

[vuillaut]

There might be no LST 1 events in there, I guess the default is aalowedtels=[1]?

yes, then I suggest to require the user explicitly an input file

agreed

[vuillaut]

Note that you could also add to the config:

"source_config": {
    "EventSource": {
        "focal_length_choice": "EQUIVALENT"
    }
}

[maxnoe]

@vuillaut no! we should always use EFFECTIVE. All our current simulations files should have the correct effective focal length specified.

[vuillaut]

@vuillaut no! we should always use EFFECTIVE. All our current simulations files should have the correct effective focal length specified.

I agree, if one wants meaningful results and if one has correct simulations files easily accessible.
However, having the possibility for users to quickly run a command and see how it behaves is also important and does not have to be 100% accurate.

[maxnoe]

However, having the possibility for users to quickly run a command and see how it behaves is also important and does not have to be 100% accurate.

But teaching people to just set an option that will result in a reconstruction bias is quite dangerous. We should replace the gamma_test_large.simtel.gz everywhere we no use it as an example or default with a better file.

We produced one some time ago following the LST simtel config, but I guess we never started to use it.

It's in the non-public test data:

test_data/mc/simtel_theta_20_az_180_gdiffuse_10evts.simtel.gz

[vuillaut]

It's in the non-public test data:

test_data/mc/simtel_theta_20_az_180_gdiffuse_10evts.simtel.gz

The non-public part for such a test file is really a drawback. What is blocking to make it public? Aren't the whole chain and settings to produce it already public?

[maxnoe]

Aren't the whole chain and settings to produce it already public?

yes

What is blocking to make it public?

Nothing in particular I guess, just that we don't have easy access to add new files to the dataserver that hosts the files that makes ctapipe.utils.get_dataset_path() work, it's a legacy system hosted by in2p3 and the ctao system that should replace it is not in place yet and I asked for a whole for an LST solution to hosting test files (both private and public) but that's also not yet setup.

[vuillaut]

Aren't the whole chain and settings to produce it already public?

yes

What is blocking to make it public?

Nothing in particular I guess, just that we don't have easy access to add new files to the dataserver that hosts the files that makes ctapipe.utils.get_dataset_path() work, it's a legacy system hosted by in2p3 and the ctao system that should replace it is not in place yet and I asked for a whole for an LST solution to hosting test files (both private and public) but that's also not yet setup.

wouldn't a simple github repo works? to be deleted once we have a more elegant solution?